2023-04-20 12:53:29,734 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8f86655c-bc81-b89c-ce6f-c96a6f801026 2023-04-20 12:53:29,745 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-04-20 12:53:29,759 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-04-20 12:53:29,760 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8f86655c-bc81-b89c-ce6f-c96a6f801026/cluster_69223e73-de8d-e90f-7d69-be39760cd8eb, deleteOnExit=true 2023-04-20 12:53:29,760 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-04-20 12:53:29,761 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8f86655c-bc81-b89c-ce6f-c96a6f801026/test.cache.data in system properties and HBase conf 2023-04-20 12:53:29,761 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8f86655c-bc81-b89c-ce6f-c96a6f801026/hadoop.tmp.dir in system properties and HBase conf 2023-04-20 12:53:29,762 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8f86655c-bc81-b89c-ce6f-c96a6f801026/hadoop.log.dir in system properties and HBase conf 2023-04-20 12:53:29,763 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8f86655c-bc81-b89c-ce6f-c96a6f801026/mapreduce.cluster.local.dir in system properties and HBase conf 2023-04-20 12:53:29,763 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8f86655c-bc81-b89c-ce6f-c96a6f801026/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-04-20 12:53:29,763 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-04-20 12:53:29,850 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-04-20 12:53:30,150 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-04-20 12:53:30,154 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8f86655c-bc81-b89c-ce6f-c96a6f801026/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-04-20 12:53:30,155 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8f86655c-bc81-b89c-ce6f-c96a6f801026/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-04-20 12:53:30,155 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8f86655c-bc81-b89c-ce6f-c96a6f801026/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-04-20 12:53:30,155 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8f86655c-bc81-b89c-ce6f-c96a6f801026/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-20 12:53:30,155 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8f86655c-bc81-b89c-ce6f-c96a6f801026/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-04-20 12:53:30,156 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8f86655c-bc81-b89c-ce6f-c96a6f801026/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-04-20 12:53:30,156 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8f86655c-bc81-b89c-ce6f-c96a6f801026/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-20 12:53:30,156 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8f86655c-bc81-b89c-ce6f-c96a6f801026/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-20 12:53:30,157 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8f86655c-bc81-b89c-ce6f-c96a6f801026/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-04-20 12:53:30,157 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8f86655c-bc81-b89c-ce6f-c96a6f801026/nfs.dump.dir in system properties and HBase conf 2023-04-20 12:53:30,157 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8f86655c-bc81-b89c-ce6f-c96a6f801026/java.io.tmpdir in system properties and HBase conf 2023-04-20 12:53:30,158 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8f86655c-bc81-b89c-ce6f-c96a6f801026/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-20 12:53:30,158 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8f86655c-bc81-b89c-ce6f-c96a6f801026/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-04-20 12:53:30,159 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8f86655c-bc81-b89c-ce6f-c96a6f801026/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-04-20 12:53:30,554 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-20 12:53:30,557 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-20 12:53:30,773 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-04-20 12:53:30,905 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-04-20 12:53:30,918 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-20 12:53:30,948 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-04-20 12:53:30,973 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8f86655c-bc81-b89c-ce6f-c96a6f801026/java.io.tmpdir/Jetty_localhost_localdomain_43273_hdfs____mrcuq1/webapp 2023-04-20 12:53:31,088 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:43273 2023-04-20 12:53:31,120 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-20 12:53:31,121 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-20 12:53:31,428 WARN [Listener at localhost.localdomain/45293] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-20 12:53:31,519 WARN [Listener at localhost.localdomain/45293] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-20 12:53:31,534 WARN [Listener at localhost.localdomain/45293] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-20 12:53:31,539 INFO [Listener at localhost.localdomain/45293] log.Slf4jLog(67): jetty-6.1.26 2023-04-20 12:53:31,544 INFO [Listener at localhost.localdomain/45293] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8f86655c-bc81-b89c-ce6f-c96a6f801026/java.io.tmpdir/Jetty_localhost_37885_datanode____.v2gjt4/webapp 2023-04-20 12:53:31,627 INFO [Listener at localhost.localdomain/45293] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:37885 2023-04-20 12:53:31,884 WARN [Listener at localhost.localdomain/42205] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-20 12:53:32,211 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x173363ad1cd6d2b0: Processing first storage report for DS-030f0aae-913b-495e-a99b-0dd57932f5c8 from datanode ceb39797-2841-41e5-8250-6d3f85306e32 2023-04-20 12:53:32,212 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x173363ad1cd6d2b0: from storage DS-030f0aae-913b-495e-a99b-0dd57932f5c8 node DatanodeRegistration(127.0.0.1:34879, datanodeUuid=ceb39797-2841-41e5-8250-6d3f85306e32, infoPort=34377, infoSecurePort=0, ipcPort=42205, storageInfo=lv=-57;cid=testClusterID;nsid=1957172543;c=1681995210614), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-04-20 12:53:32,212 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x173363ad1cd6d2b0: Processing first storage report for DS-d6234183-7845-4cbd-8e54-cecc650209ba from datanode ceb39797-2841-41e5-8250-6d3f85306e32 2023-04-20 12:53:32,212 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x173363ad1cd6d2b0: from storage DS-d6234183-7845-4cbd-8e54-cecc650209ba node DatanodeRegistration(127.0.0.1:34879, datanodeUuid=ceb39797-2841-41e5-8250-6d3f85306e32, infoPort=34377, infoSecurePort=0, ipcPort=42205, storageInfo=lv=-57;cid=testClusterID;nsid=1957172543;c=1681995210614), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-04-20 12:53:32,229 DEBUG [Listener at localhost.localdomain/42205] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8f86655c-bc81-b89c-ce6f-c96a6f801026 2023-04-20 12:53:32,289 INFO [Listener at localhost.localdomain/42205] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8f86655c-bc81-b89c-ce6f-c96a6f801026/cluster_69223e73-de8d-e90f-7d69-be39760cd8eb/zookeeper_0, clientPort=56673, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8f86655c-bc81-b89c-ce6f-c96a6f801026/cluster_69223e73-de8d-e90f-7d69-be39760cd8eb/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8f86655c-bc81-b89c-ce6f-c96a6f801026/cluster_69223e73-de8d-e90f-7d69-be39760cd8eb/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-04-20 12:53:32,300 INFO [Listener at localhost.localdomain/42205] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=56673 2023-04-20 12:53:32,310 INFO [Listener at localhost.localdomain/42205] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-20 12:53:32,314 INFO [Listener at localhost.localdomain/42205] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-20 12:53:32,920 INFO [Listener at localhost.localdomain/42205] util.FSUtils(471): Created version file at hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd with version=8 2023-04-20 12:53:32,920 INFO [Listener at localhost.localdomain/42205] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/hbase-staging 2023-04-20 12:53:33,159 INFO [Listener at localhost.localdomain/42205] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-20 12:53:33,512 INFO [Listener at localhost.localdomain/42205] client.ConnectionUtils(127): master/jenkins-hbase20:0 server-side Connection retries=45 2023-04-20 12:53:33,537 INFO [Listener at localhost.localdomain/42205] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-20 12:53:33,538 INFO [Listener at localhost.localdomain/42205] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-20 12:53:33,538 INFO [Listener at localhost.localdomain/42205] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-20 12:53:33,538 INFO [Listener at localhost.localdomain/42205] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-20 12:53:33,538 INFO [Listener at localhost.localdomain/42205] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-20 12:53:33,648 INFO [Listener at localhost.localdomain/42205] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-04-20 12:53:33,705 DEBUG [Listener at localhost.localdomain/42205] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-04-20 12:53:33,780 INFO [Listener at localhost.localdomain/42205] ipc.NettyRpcServer(120): Bind to /148.251.75.209:40401 2023-04-20 12:53:33,789 INFO [Listener at localhost.localdomain/42205] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-20 12:53:33,791 INFO [Listener at localhost.localdomain/42205] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-20 12:53:33,809 INFO [Listener at localhost.localdomain/42205] zookeeper.RecoverableZooKeeper(93): Process identifier=master:40401 connecting to ZooKeeper ensemble=127.0.0.1:56673 2023-04-20 12:53:33,841 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:404010x0, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-20 12:53:33,844 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:40401-0x100ce703e210000 connected 2023-04-20 12:53:33,868 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-20 12:53:33,869 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-20 12:53:33,872 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-20 12:53:33,881 DEBUG [Listener at localhost.localdomain/42205] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=40401 2023-04-20 12:53:33,882 DEBUG [Listener at localhost.localdomain/42205] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=40401 2023-04-20 12:53:33,885 DEBUG [Listener at localhost.localdomain/42205] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=40401 2023-04-20 12:53:33,886 DEBUG [Listener at localhost.localdomain/42205] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=40401 2023-04-20 12:53:33,887 DEBUG [Listener at localhost.localdomain/42205] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=40401 2023-04-20 12:53:33,891 INFO [Listener at localhost.localdomain/42205] master.HMaster(444): hbase.rootdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd, hbase.cluster.distributed=false 2023-04-20 12:53:33,947 INFO [Listener at localhost.localdomain/42205] client.ConnectionUtils(127): regionserver/jenkins-hbase20:0 server-side Connection retries=45 2023-04-20 12:53:33,948 INFO [Listener at localhost.localdomain/42205] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-20 12:53:33,948 INFO [Listener at localhost.localdomain/42205] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-20 12:53:33,948 INFO [Listener at localhost.localdomain/42205] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-20 12:53:33,948 INFO [Listener at localhost.localdomain/42205] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-20 12:53:33,948 INFO [Listener at localhost.localdomain/42205] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-20 12:53:33,952 INFO [Listener at localhost.localdomain/42205] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-20 12:53:33,955 INFO [Listener at localhost.localdomain/42205] ipc.NettyRpcServer(120): Bind to /148.251.75.209:36277 2023-04-20 12:53:33,957 INFO [Listener at localhost.localdomain/42205] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-20 12:53:33,962 DEBUG [Listener at localhost.localdomain/42205] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-20 12:53:33,962 INFO [Listener at localhost.localdomain/42205] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-20 12:53:33,965 INFO [Listener at localhost.localdomain/42205] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-20 12:53:33,966 INFO [Listener at localhost.localdomain/42205] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:36277 connecting to ZooKeeper ensemble=127.0.0.1:56673 2023-04-20 12:53:33,970 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:362770x0, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-20 12:53:33,971 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:36277-0x100ce703e210001 connected 2023-04-20 12:53:33,972 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-20 12:53:33,973 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-20 12:53:33,974 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-20 12:53:33,974 DEBUG [Listener at localhost.localdomain/42205] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=36277 2023-04-20 12:53:33,974 DEBUG [Listener at localhost.localdomain/42205] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=36277 2023-04-20 12:53:33,975 DEBUG [Listener at localhost.localdomain/42205] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=36277 2023-04-20 12:53:33,975 DEBUG [Listener at localhost.localdomain/42205] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=36277 2023-04-20 12:53:33,975 DEBUG [Listener at localhost.localdomain/42205] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=36277 2023-04-20 12:53:33,977 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase20.apache.org,40401,1681995213032 2023-04-20 12:53:33,985 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-20 12:53:33,987 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase20.apache.org,40401,1681995213032 2023-04-20 12:53:34,004 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-20 12:53:34,004 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-20 12:53:34,004 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-20 12:53:34,005 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-20 12:53:34,006 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase20.apache.org,40401,1681995213032 from backup master directory 2023-04-20 12:53:34,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-20 12:53:34,009 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase20.apache.org,40401,1681995213032 2023-04-20 12:53:34,009 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-20 12:53:34,009 WARN [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-20 12:53:34,009 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase20.apache.org,40401,1681995213032 2023-04-20 12:53:34,012 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-04-20 12:53:34,013 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-04-20 12:53:34,090 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/hbase.id with ID: beaafe66-dec3-443c-bbde-079e253bd039 2023-04-20 12:53:34,139 INFO [master/jenkins-hbase20:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-20 12:53:34,151 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-20 12:53:34,191 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x38302e48 to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:53:34,219 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1c892a08, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:53:34,238 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-20 12:53:34,240 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-04-20 12:53:34,254 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-04-20 12:53:34,255 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-04-20 12:53:34,256 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-20 12:53:34,259 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-20 12:53:34,260 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-20 12:53:34,286 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/MasterData/data/master/store-tmp 2023-04-20 12:53:34,311 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-20 12:53:34,311 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-20 12:53:34,311 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-20 12:53:34,312 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-20 12:53:34,312 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-20 12:53:34,312 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-20 12:53:34,312 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-20 12:53:34,312 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-20 12:53:34,314 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/MasterData/WALs/jenkins-hbase20.apache.org,40401,1681995213032 2023-04-20 12:53:34,335 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C40401%2C1681995213032, suffix=, logDir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/MasterData/WALs/jenkins-hbase20.apache.org,40401,1681995213032, archiveDir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/MasterData/oldWALs, maxLogs=10 2023-04-20 12:53:34,379 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34879,DS-030f0aae-913b-495e-a99b-0dd57932f5c8,DISK] 2023-04-20 12:53:34,386 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-20 12:53:34,436 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/MasterData/WALs/jenkins-hbase20.apache.org,40401,1681995213032/jenkins-hbase20.apache.org%2C40401%2C1681995213032.1681995214343 2023-04-20 12:53:34,436 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34879,DS-030f0aae-913b-495e-a99b-0dd57932f5c8,DISK]] 2023-04-20 12:53:34,437 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-04-20 12:53:34,437 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-20 12:53:34,440 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-04-20 12:53:34,440 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-04-20 12:53:34,493 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-04-20 12:53:34,500 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-04-20 12:53:34,519 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-04-20 12:53:34,531 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 12:53:34,536 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-20 12:53:34,538 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-20 12:53:34,553 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-04-20 12:53:34,557 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-20 12:53:34,559 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59660128, jitterRate=-0.11099481582641602}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-20 12:53:34,559 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-20 12:53:34,560 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-04-20 12:53:34,579 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-04-20 12:53:34,579 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-04-20 12:53:34,581 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-20 12:53:34,583 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-20 12:53:34,608 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 25 msec 2023-04-20 12:53:34,608 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-04-20 12:53:34,634 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-04-20 12:53:34,639 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-04-20 12:53:34,661 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-04-20 12:53:34,664 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-20 12:53:34,666 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-04-20 12:53:34,671 INFO [master/jenkins-hbase20:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-04-20 12:53:34,675 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-04-20 12:53:34,677 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-20 12:53:34,679 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-04-20 12:53:34,679 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-04-20 12:53:34,690 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-04-20 12:53:34,693 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-20 12:53:34,693 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-20 12:53:34,694 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-20 12:53:34,694 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase20.apache.org,40401,1681995213032, sessionid=0x100ce703e210000, setting cluster-up flag (Was=false) 2023-04-20 12:53:34,710 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-20 12:53:34,714 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-04-20 12:53:34,716 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,40401,1681995213032 2023-04-20 12:53:34,730 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-20 12:53:34,734 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-04-20 12:53:34,735 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,40401,1681995213032 2023-04-20 12:53:34,737 WARN [master/jenkins-hbase20:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.hbase-snapshot/.tmp 2023-04-20 12:53:34,779 INFO [RS:0;jenkins-hbase20:36277] regionserver.HRegionServer(952): ClusterId : beaafe66-dec3-443c-bbde-079e253bd039 2023-04-20 12:53:34,783 DEBUG [RS:0;jenkins-hbase20:36277] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-20 12:53:34,787 DEBUG [RS:0;jenkins-hbase20:36277] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-20 12:53:34,787 DEBUG [RS:0;jenkins-hbase20:36277] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-20 12:53:34,790 DEBUG [RS:0;jenkins-hbase20:36277] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-20 12:53:34,791 DEBUG [RS:0;jenkins-hbase20:36277] zookeeper.ReadOnlyZKClient(139): Connect 0x2a9f12d4 to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:53:34,795 DEBUG [RS:0;jenkins-hbase20:36277] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2458f9f7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:53:34,796 DEBUG [RS:0;jenkins-hbase20:36277] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@206779a4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-04-20 12:53:34,816 DEBUG [RS:0;jenkins-hbase20:36277] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase20:36277 2023-04-20 12:53:34,820 INFO [RS:0;jenkins-hbase20:36277] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-20 12:53:34,820 INFO [RS:0;jenkins-hbase20:36277] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-20 12:53:34,820 DEBUG [RS:0;jenkins-hbase20:36277] regionserver.HRegionServer(1023): About to register with Master. 2023-04-20 12:53:34,822 INFO [RS:0;jenkins-hbase20:36277] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase20.apache.org,40401,1681995213032 with isa=jenkins-hbase20.apache.org/148.251.75.209:36277, startcode=1681995213947 2023-04-20 12:53:34,837 DEBUG [RS:0;jenkins-hbase20:36277] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-20 12:53:34,848 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-04-20 12:53:34,857 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-04-20 12:53:34,857 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-04-20 12:53:34,858 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-04-20 12:53:34,858 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-04-20 12:53:34,858 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase20:0, corePoolSize=10, maxPoolSize=10 2023-04-20 12:53:34,858 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-20 12:53:34,858 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-04-20 12:53:34,858 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-20 12:53:34,863 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681995244863 2023-04-20 12:53:34,865 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-04-20 12:53:34,867 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-04-20 12:53:34,868 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-04-20 12:53:34,872 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-20 12:53:34,879 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-04-20 12:53:34,886 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-04-20 12:53:34,887 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-04-20 12:53:34,887 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-04-20 12:53:34,888 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-04-20 12:53:34,888 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:38403, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-04-20 12:53:34,889 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-20 12:53:34,890 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-04-20 12:53:34,895 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-04-20 12:53:34,896 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-04-20 12:53:34,898 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-04-20 12:53:34,900 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-04-20 12:53:34,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:34,901 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1681995214901,5,FailOnTimeoutGroup] 2023-04-20 12:53:34,902 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1681995214901,5,FailOnTimeoutGroup] 2023-04-20 12:53:34,902 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-20 12:53:34,902 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-04-20 12:53:34,903 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-04-20 12:53:34,903 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-04-20 12:53:34,922 DEBUG [RS:0;jenkins-hbase20:36277] regionserver.HRegionServer(2829): Master is not running yet 2023-04-20 12:53:34,923 WARN [RS:0;jenkins-hbase20:36277] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-04-20 12:53:35,025 INFO [RS:0;jenkins-hbase20:36277] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase20.apache.org,40401,1681995213032 with isa=jenkins-hbase20.apache.org/148.251.75.209:36277, startcode=1681995213947 2023-04-20 12:53:35,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:35,041 DEBUG [RS:0;jenkins-hbase20:36277] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd 2023-04-20 12:53:35,041 DEBUG [RS:0;jenkins-hbase20:36277] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost.localdomain:45293 2023-04-20 12:53:35,042 DEBUG [RS:0;jenkins-hbase20:36277] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-20 12:53:35,046 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-20 12:53:35,047 DEBUG [RS:0;jenkins-hbase20:36277] zookeeper.ZKUtil(162): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:35,047 WARN [RS:0;jenkins-hbase20:36277] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-20 12:53:35,047 INFO [RS:0;jenkins-hbase20:36277] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-20 12:53:35,048 DEBUG [RS:0;jenkins-hbase20:36277] regionserver.HRegionServer(1947): logDir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/WALs/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:35,048 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase20.apache.org,36277,1681995213947] 2023-04-20 12:53:35,058 DEBUG [RS:0;jenkins-hbase20:36277] zookeeper.ZKUtil(162): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:35,071 DEBUG [RS:0;jenkins-hbase20:36277] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-20 12:53:35,079 INFO [RS:0;jenkins-hbase20:36277] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-20 12:53:35,095 INFO [RS:0;jenkins-hbase20:36277] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-20 12:53:35,098 INFO [RS:0;jenkins-hbase20:36277] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-20 12:53:35,098 INFO [RS:0;jenkins-hbase20:36277] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-20 12:53:35,099 INFO [RS:0;jenkins-hbase20:36277] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-20 12:53:35,104 INFO [RS:0;jenkins-hbase20:36277] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-20 12:53:35,105 DEBUG [RS:0;jenkins-hbase20:36277] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-20 12:53:35,105 DEBUG [RS:0;jenkins-hbase20:36277] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-20 12:53:35,105 DEBUG [RS:0;jenkins-hbase20:36277] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-20 12:53:35,105 DEBUG [RS:0;jenkins-hbase20:36277] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-20 12:53:35,105 DEBUG [RS:0;jenkins-hbase20:36277] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-20 12:53:35,105 DEBUG [RS:0;jenkins-hbase20:36277] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-04-20 12:53:35,106 DEBUG [RS:0;jenkins-hbase20:36277] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-20 12:53:35,106 DEBUG [RS:0;jenkins-hbase20:36277] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-20 12:53:35,106 DEBUG [RS:0;jenkins-hbase20:36277] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-20 12:53:35,106 DEBUG [RS:0;jenkins-hbase20:36277] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-20 12:53:35,107 INFO [RS:0;jenkins-hbase20:36277] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-20 12:53:35,107 INFO [RS:0;jenkins-hbase20:36277] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-20 12:53:35,107 INFO [RS:0;jenkins-hbase20:36277] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-20 12:53:35,119 INFO [RS:0;jenkins-hbase20:36277] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-20 12:53:35,121 INFO [RS:0;jenkins-hbase20:36277] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,36277,1681995213947-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-20 12:53:35,134 INFO [RS:0;jenkins-hbase20:36277] regionserver.Replication(203): jenkins-hbase20.apache.org,36277,1681995213947 started 2023-04-20 12:53:35,134 INFO [RS:0;jenkins-hbase20:36277] regionserver.HRegionServer(1638): Serving as jenkins-hbase20.apache.org,36277,1681995213947, RpcServer on jenkins-hbase20.apache.org/148.251.75.209:36277, sessionid=0x100ce703e210001 2023-04-20 12:53:35,134 DEBUG [RS:0;jenkins-hbase20:36277] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-20 12:53:35,135 DEBUG [RS:0;jenkins-hbase20:36277] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:35,135 DEBUG [RS:0;jenkins-hbase20:36277] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,36277,1681995213947' 2023-04-20 12:53:35,135 DEBUG [RS:0;jenkins-hbase20:36277] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:35,135 DEBUG [RS:0;jenkins-hbase20:36277] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:35,136 DEBUG [RS:0;jenkins-hbase20:36277] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-20 12:53:35,136 DEBUG [RS:0;jenkins-hbase20:36277] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-20 12:53:35,136 DEBUG [RS:0;jenkins-hbase20:36277] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:35,136 DEBUG [RS:0;jenkins-hbase20:36277] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,36277,1681995213947' 2023-04-20 12:53:35,136 DEBUG [RS:0;jenkins-hbase20:36277] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-20 12:53:35,137 DEBUG [RS:0;jenkins-hbase20:36277] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-20 12:53:35,137 DEBUG [RS:0;jenkins-hbase20:36277] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-20 12:53:35,137 INFO [RS:0;jenkins-hbase20:36277] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-20 12:53:35,137 INFO [RS:0;jenkins-hbase20:36277] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-20 12:53:35,252 INFO [RS:0;jenkins-hbase20:36277] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C36277%2C1681995213947, suffix=, logDir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/WALs/jenkins-hbase20.apache.org,36277,1681995213947, archiveDir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/oldWALs, maxLogs=32 2023-04-20 12:53:35,267 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34879,DS-030f0aae-913b-495e-a99b-0dd57932f5c8,DISK] 2023-04-20 12:53:35,270 INFO [RS:0;jenkins-hbase20:36277] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/WALs/jenkins-hbase20.apache.org,36277,1681995213947/jenkins-hbase20.apache.org%2C36277%2C1681995213947.1681995215254 2023-04-20 12:53:35,270 DEBUG [RS:0;jenkins-hbase20:36277] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34879,DS-030f0aae-913b-495e-a99b-0dd57932f5c8,DISK]] 2023-04-20 12:53:35,301 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-20 12:53:35,303 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-20 12:53:35,303 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd 2023-04-20 12:53:35,317 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-20 12:53:35,319 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-20 12:53:35,321 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/hbase/meta/1588230740/info 2023-04-20 12:53:35,322 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-20 12:53:35,323 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 12:53:35,323 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-20 12:53:35,326 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/hbase/meta/1588230740/rep_barrier 2023-04-20 12:53:35,326 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-20 12:53:35,327 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 12:53:35,327 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-20 12:53:35,329 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/hbase/meta/1588230740/table 2023-04-20 12:53:35,330 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-20 12:53:35,331 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 12:53:35,332 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/hbase/meta/1588230740 2023-04-20 12:53:35,333 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/hbase/meta/1588230740 2023-04-20 12:53:35,337 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-20 12:53:35,338 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-20 12:53:35,341 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-20 12:53:35,342 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74370385, jitterRate=0.10820509493350983}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-20 12:53:35,342 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-20 12:53:35,343 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-20 12:53:35,343 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-20 12:53:35,343 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-20 12:53:35,343 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-20 12:53:35,343 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-20 12:53:35,344 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-20 12:53:35,344 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-20 12:53:35,351 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-04-20 12:53:35,351 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-04-20 12:53:35,359 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-04-20 12:53:35,370 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-04-20 12:53:35,372 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-20 12:53:35,527 DEBUG [jenkins-hbase20:40401] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-04-20 12:53:35,530 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,36277,1681995213947, state=OPENING 2023-04-20 12:53:35,539 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-04-20 12:53:35,540 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-20 12:53:35,541 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-20 12:53:35,543 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,36277,1681995213947}] 2023-04-20 12:53:35,725 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:35,727 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-20 12:53:35,730 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:60940, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-20 12:53:35,741 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-04-20 12:53:35,742 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-20 12:53:35,746 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C36277%2C1681995213947.meta, suffix=.meta, logDir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/WALs/jenkins-hbase20.apache.org,36277,1681995213947, archiveDir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/oldWALs, maxLogs=32 2023-04-20 12:53:35,764 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34879,DS-030f0aae-913b-495e-a99b-0dd57932f5c8,DISK] 2023-04-20 12:53:35,768 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/WALs/jenkins-hbase20.apache.org,36277,1681995213947/jenkins-hbase20.apache.org%2C36277%2C1681995213947.meta.1681995215748.meta 2023-04-20 12:53:35,769 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34879,DS-030f0aae-913b-495e-a99b-0dd57932f5c8,DISK]] 2023-04-20 12:53:35,769 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-04-20 12:53:35,771 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-04-20 12:53:35,790 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-04-20 12:53:35,794 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-04-20 12:53:35,799 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-04-20 12:53:35,799 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-20 12:53:35,799 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-04-20 12:53:35,799 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-04-20 12:53:35,802 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-20 12:53:35,804 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/hbase/meta/1588230740/info 2023-04-20 12:53:35,804 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/hbase/meta/1588230740/info 2023-04-20 12:53:35,805 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-20 12:53:35,806 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 12:53:35,806 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-20 12:53:35,808 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/hbase/meta/1588230740/rep_barrier 2023-04-20 12:53:35,808 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/hbase/meta/1588230740/rep_barrier 2023-04-20 12:53:35,809 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-20 12:53:35,810 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 12:53:35,810 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-20 12:53:35,811 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/hbase/meta/1588230740/table 2023-04-20 12:53:35,811 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/hbase/meta/1588230740/table 2023-04-20 12:53:35,812 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-20 12:53:35,813 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 12:53:35,815 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/hbase/meta/1588230740 2023-04-20 12:53:35,818 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/hbase/meta/1588230740 2023-04-20 12:53:35,821 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-20 12:53:35,823 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-20 12:53:35,825 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72323619, jitterRate=0.07770590484142303}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-20 12:53:35,825 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-20 12:53:35,835 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1681995215719 2023-04-20 12:53:35,850 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-04-20 12:53:35,851 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-04-20 12:53:35,851 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,36277,1681995213947, state=OPEN 2023-04-20 12:53:35,853 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-04-20 12:53:35,853 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-20 12:53:35,857 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-04-20 12:53:35,857 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,36277,1681995213947 in 311 msec 2023-04-20 12:53:35,862 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-04-20 12:53:35,862 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 499 msec 2023-04-20 12:53:35,867 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.0880 sec 2023-04-20 12:53:35,867 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1681995215867, completionTime=-1 2023-04-20 12:53:35,868 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-04-20 12:53:35,868 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-04-20 12:53:35,923 DEBUG [hconnection-0x58166cba-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:53:35,926 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:60950, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:53:35,942 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-04-20 12:53:35,942 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681995275942 2023-04-20 12:53:35,942 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681995335942 2023-04-20 12:53:35,942 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 74 msec 2023-04-20 12:53:35,965 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,40401,1681995213032-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-20 12:53:35,966 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,40401,1681995213032-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-20 12:53:35,966 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,40401,1681995213032-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-20 12:53:35,967 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase20:40401, period=300000, unit=MILLISECONDS is enabled. 2023-04-20 12:53:35,968 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-04-20 12:53:35,974 DEBUG [master/jenkins-hbase20:0.Chore.1] janitor.CatalogJanitor(175): 2023-04-20 12:53:35,982 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-04-20 12:53:35,983 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-20 12:53:35,990 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-04-20 12:53:35,993 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-04-20 12:53:35,996 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-20 12:53:36,015 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/hbase/namespace/4b9689ab36fe7756d2951e73df110950 2023-04-20 12:53:36,018 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/hbase/namespace/4b9689ab36fe7756d2951e73df110950 empty. 2023-04-20 12:53:36,019 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/hbase/namespace/4b9689ab36fe7756d2951e73df110950 2023-04-20 12:53:36,019 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-04-20 12:53:36,451 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-04-20 12:53:36,455 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 4b9689ab36fe7756d2951e73df110950, NAME => 'hbase:namespace,,1681995215982.4b9689ab36fe7756d2951e73df110950.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp 2023-04-20 12:53:36,470 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1681995215982.4b9689ab36fe7756d2951e73df110950.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-20 12:53:36,470 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 4b9689ab36fe7756d2951e73df110950, disabling compactions & flushes 2023-04-20 12:53:36,470 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1681995215982.4b9689ab36fe7756d2951e73df110950. 2023-04-20 12:53:36,470 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1681995215982.4b9689ab36fe7756d2951e73df110950. 2023-04-20 12:53:36,470 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1681995215982.4b9689ab36fe7756d2951e73df110950. after waiting 0 ms 2023-04-20 12:53:36,470 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1681995215982.4b9689ab36fe7756d2951e73df110950. 2023-04-20 12:53:36,471 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1681995215982.4b9689ab36fe7756d2951e73df110950. 2023-04-20 12:53:36,471 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 4b9689ab36fe7756d2951e73df110950: 2023-04-20 12:53:36,476 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-04-20 12:53:36,492 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1681995215982.4b9689ab36fe7756d2951e73df110950.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681995216478"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681995216478"}]},"ts":"1681995216478"} 2023-04-20 12:53:36,515 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-20 12:53:36,516 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-20 12:53:36,521 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681995216517"}]},"ts":"1681995216517"} 2023-04-20 12:53:36,525 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-04-20 12:53:36,532 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=4b9689ab36fe7756d2951e73df110950, ASSIGN}] 2023-04-20 12:53:36,536 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=4b9689ab36fe7756d2951e73df110950, ASSIGN 2023-04-20 12:53:36,538 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=4b9689ab36fe7756d2951e73df110950, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,36277,1681995213947; forceNewPlan=false, retain=false 2023-04-20 12:53:36,689 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=4b9689ab36fe7756d2951e73df110950, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:36,689 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1681995215982.4b9689ab36fe7756d2951e73df110950.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681995216689"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681995216689"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681995216689"}]},"ts":"1681995216689"} 2023-04-20 12:53:36,696 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 4b9689ab36fe7756d2951e73df110950, server=jenkins-hbase20.apache.org,36277,1681995213947}] 2023-04-20 12:53:36,864 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1681995215982.4b9689ab36fe7756d2951e73df110950. 2023-04-20 12:53:36,865 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 4b9689ab36fe7756d2951e73df110950, NAME => 'hbase:namespace,,1681995215982.4b9689ab36fe7756d2951e73df110950.', STARTKEY => '', ENDKEY => ''} 2023-04-20 12:53:36,866 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 4b9689ab36fe7756d2951e73df110950 2023-04-20 12:53:36,866 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1681995215982.4b9689ab36fe7756d2951e73df110950.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-20 12:53:36,866 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 4b9689ab36fe7756d2951e73df110950 2023-04-20 12:53:36,866 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 4b9689ab36fe7756d2951e73df110950 2023-04-20 12:53:36,868 INFO [StoreOpener-4b9689ab36fe7756d2951e73df110950-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 4b9689ab36fe7756d2951e73df110950 2023-04-20 12:53:36,870 DEBUG [StoreOpener-4b9689ab36fe7756d2951e73df110950-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/hbase/namespace/4b9689ab36fe7756d2951e73df110950/info 2023-04-20 12:53:36,870 DEBUG [StoreOpener-4b9689ab36fe7756d2951e73df110950-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/hbase/namespace/4b9689ab36fe7756d2951e73df110950/info 2023-04-20 12:53:36,871 INFO [StoreOpener-4b9689ab36fe7756d2951e73df110950-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4b9689ab36fe7756d2951e73df110950 columnFamilyName info 2023-04-20 12:53:36,872 INFO [StoreOpener-4b9689ab36fe7756d2951e73df110950-1] regionserver.HStore(310): Store=4b9689ab36fe7756d2951e73df110950/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 12:53:36,873 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/hbase/namespace/4b9689ab36fe7756d2951e73df110950 2023-04-20 12:53:36,874 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/hbase/namespace/4b9689ab36fe7756d2951e73df110950 2023-04-20 12:53:36,879 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 4b9689ab36fe7756d2951e73df110950 2023-04-20 12:53:36,883 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/hbase/namespace/4b9689ab36fe7756d2951e73df110950/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-20 12:53:36,884 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 4b9689ab36fe7756d2951e73df110950; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60838857, jitterRate=-0.09343038499355316}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-20 12:53:36,884 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 4b9689ab36fe7756d2951e73df110950: 2023-04-20 12:53:36,886 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1681995215982.4b9689ab36fe7756d2951e73df110950., pid=6, masterSystemTime=1681995216852 2023-04-20 12:53:36,889 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1681995215982.4b9689ab36fe7756d2951e73df110950. 2023-04-20 12:53:36,889 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1681995215982.4b9689ab36fe7756d2951e73df110950. 2023-04-20 12:53:36,890 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=4b9689ab36fe7756d2951e73df110950, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:36,891 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1681995215982.4b9689ab36fe7756d2951e73df110950.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681995216890"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1681995216890"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681995216890"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681995216890"}]},"ts":"1681995216890"} 2023-04-20 12:53:36,897 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-04-20 12:53:36,897 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 4b9689ab36fe7756d2951e73df110950, server=jenkins-hbase20.apache.org,36277,1681995213947 in 198 msec 2023-04-20 12:53:36,901 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-04-20 12:53:36,901 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=4b9689ab36fe7756d2951e73df110950, ASSIGN in 365 msec 2023-04-20 12:53:36,902 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-20 12:53:36,903 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681995216903"}]},"ts":"1681995216903"} 2023-04-20 12:53:36,905 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-04-20 12:53:36,908 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-04-20 12:53:36,911 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 925 msec 2023-04-20 12:53:36,995 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-04-20 12:53:36,998 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-04-20 12:53:36,998 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-20 12:53:37,040 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-04-20 12:53:37,057 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-20 12:53:37,063 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 32 msec 2023-04-20 12:53:37,075 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-04-20 12:53:37,088 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-20 12:53:37,092 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 17 msec 2023-04-20 12:53:37,109 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-04-20 12:53:37,111 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-04-20 12:53:37,113 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.103sec 2023-04-20 12:53:37,115 INFO [master/jenkins-hbase20:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-04-20 12:53:37,117 INFO [master/jenkins-hbase20:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-04-20 12:53:37,118 INFO [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-04-20 12:53:37,119 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,40401,1681995213032-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-04-20 12:53:37,119 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,40401,1681995213032-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-04-20 12:53:37,131 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-04-20 12:53:37,185 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x4c9bc423 to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:53:37,190 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7854c900, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:53:37,202 DEBUG [hconnection-0xac2f484-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:53:37,213 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:60956, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:53:37,225 INFO [Listener at localhost.localdomain/42205] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase20.apache.org,40401,1681995213032 2023-04-20 12:53:37,250 INFO [Listener at localhost.localdomain/42205] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=207, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=298, ProcessCount=172, AvailableMemoryMB=9803 2023-04-20 12:53:37,269 DEBUG [Listener at localhost.localdomain/42205] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-20 12:53:37,272 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52006, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-20 12:53:37,281 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-20 12:53:37,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-20 12:53:37,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-20 12:53:37,290 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-20 12:53:37,292 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-20 12:53:37,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-04-20 12:53:37,295 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036 2023-04-20 12:53:37,297 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036 empty. 2023-04-20 12:53:37,297 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036 2023-04-20 12:53:37,297 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-20 12:53:37,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-20 12:53:37,318 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-20 12:53:37,320 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => fc3812e05598851193449f5684adf036, NAME => 'TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp 2023-04-20 12:53:37,335 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-20 12:53:37,335 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing fc3812e05598851193449f5684adf036, disabling compactions & flushes 2023-04-20 12:53:37,335 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:37,336 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:37,336 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. after waiting 0 ms 2023-04-20 12:53:37,336 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:37,336 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:37,336 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:37,341 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-20 12:53:37,343 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681995217342"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681995217342"}]},"ts":"1681995217342"} 2023-04-20 12:53:37,346 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-20 12:53:37,348 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-20 12:53:37,348 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681995217348"}]},"ts":"1681995217348"} 2023-04-20 12:53:37,351 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-20 12:53:37,354 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=fc3812e05598851193449f5684adf036, ASSIGN}] 2023-04-20 12:53:37,357 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=fc3812e05598851193449f5684adf036, ASSIGN 2023-04-20 12:53:37,359 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=fc3812e05598851193449f5684adf036, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,36277,1681995213947; forceNewPlan=false, retain=false 2023-04-20 12:53:37,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-20 12:53:37,510 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=fc3812e05598851193449f5684adf036, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:37,511 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681995217510"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681995217510"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681995217510"}]},"ts":"1681995217510"} 2023-04-20 12:53:37,514 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947}] 2023-04-20 12:53:37,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-20 12:53:37,675 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:37,675 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => fc3812e05598851193449f5684adf036, NAME => 'TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.', STARTKEY => '', ENDKEY => ''} 2023-04-20 12:53:37,676 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees fc3812e05598851193449f5684adf036 2023-04-20 12:53:37,676 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-20 12:53:37,676 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for fc3812e05598851193449f5684adf036 2023-04-20 12:53:37,676 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for fc3812e05598851193449f5684adf036 2023-04-20 12:53:37,678 INFO [StoreOpener-fc3812e05598851193449f5684adf036-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region fc3812e05598851193449f5684adf036 2023-04-20 12:53:37,681 DEBUG [StoreOpener-fc3812e05598851193449f5684adf036-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A 2023-04-20 12:53:37,681 DEBUG [StoreOpener-fc3812e05598851193449f5684adf036-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A 2023-04-20 12:53:37,682 INFO [StoreOpener-fc3812e05598851193449f5684adf036-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 12:53:37,683 INFO [StoreOpener-fc3812e05598851193449f5684adf036-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fc3812e05598851193449f5684adf036 columnFamilyName A 2023-04-20 12:53:37,684 INFO [StoreOpener-fc3812e05598851193449f5684adf036-1] regionserver.HStore(310): Store=fc3812e05598851193449f5684adf036/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 12:53:37,684 INFO [StoreOpener-fc3812e05598851193449f5684adf036-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region fc3812e05598851193449f5684adf036 2023-04-20 12:53:37,687 DEBUG [StoreOpener-fc3812e05598851193449f5684adf036-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B 2023-04-20 12:53:37,687 DEBUG [StoreOpener-fc3812e05598851193449f5684adf036-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B 2023-04-20 12:53:37,687 INFO [StoreOpener-fc3812e05598851193449f5684adf036-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 12:53:37,688 INFO [StoreOpener-fc3812e05598851193449f5684adf036-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fc3812e05598851193449f5684adf036 columnFamilyName B 2023-04-20 12:53:37,689 INFO [StoreOpener-fc3812e05598851193449f5684adf036-1] regionserver.HStore(310): Store=fc3812e05598851193449f5684adf036/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 12:53:37,689 INFO [StoreOpener-fc3812e05598851193449f5684adf036-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region fc3812e05598851193449f5684adf036 2023-04-20 12:53:37,691 DEBUG [StoreOpener-fc3812e05598851193449f5684adf036-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C 2023-04-20 12:53:37,691 DEBUG [StoreOpener-fc3812e05598851193449f5684adf036-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C 2023-04-20 12:53:37,692 INFO [StoreOpener-fc3812e05598851193449f5684adf036-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 12:53:37,692 INFO [StoreOpener-fc3812e05598851193449f5684adf036-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fc3812e05598851193449f5684adf036 columnFamilyName C 2023-04-20 12:53:37,693 INFO [StoreOpener-fc3812e05598851193449f5684adf036-1] regionserver.HStore(310): Store=fc3812e05598851193449f5684adf036/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 12:53:37,693 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:37,696 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036 2023-04-20 12:53:37,696 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036 2023-04-20 12:53:37,700 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-20 12:53:37,702 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for fc3812e05598851193449f5684adf036 2023-04-20 12:53:37,707 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-20 12:53:37,708 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened fc3812e05598851193449f5684adf036; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60913290, jitterRate=-0.0923212468624115}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-20 12:53:37,708 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:37,709 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036., pid=11, masterSystemTime=1681995217668 2023-04-20 12:53:37,712 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:37,712 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:37,713 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=fc3812e05598851193449f5684adf036, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:37,714 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681995217713"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1681995217713"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681995217713"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681995217713"}]},"ts":"1681995217713"} 2023-04-20 12:53:37,720 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-04-20 12:53:37,721 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 in 203 msec 2023-04-20 12:53:37,724 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-04-20 12:53:37,725 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=fc3812e05598851193449f5684adf036, ASSIGN in 367 msec 2023-04-20 12:53:37,726 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-20 12:53:37,726 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681995217726"}]},"ts":"1681995217726"} 2023-04-20 12:53:37,728 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-20 12:53:37,731 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-20 12:53:37,733 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 446 msec 2023-04-20 12:53:37,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-20 12:53:37,921 INFO [Listener at localhost.localdomain/42205] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-04-20 12:53:37,930 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x21c17a8e to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:53:37,936 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@537a70b3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:53:37,939 DEBUG [Listener at localhost.localdomain/42205] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:53:37,947 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:60964, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:53:37,955 DEBUG [Listener at localhost.localdomain/42205] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-20 12:53:37,958 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52022, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-20 12:53:37,968 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x7d6b0f87 to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:53:37,972 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2398b790, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:53:37,973 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x5d114eb9 to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:53:37,981 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1ee17908, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:53:37,981 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x3627406c to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:53:37,989 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@554a753e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:53:37,989 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x04d5a66f to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:53:37,994 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@e18c51d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:53:37,994 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x542f272a to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:53:38,006 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@31222f6c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:53:38,007 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x21551c39 to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:53:38,014 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1232cedd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:53:38,015 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x7abbc496 to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:53:38,043 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7386ab2c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:53:38,045 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x373daaaa to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:53:38,051 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7124b8aa, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:53:38,052 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x11a028ee to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:53:38,060 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4b12b88a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:53:38,066 DEBUG [hconnection-0x54820470-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:53:38,073 DEBUG [hconnection-0x2d45398-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:53:38,074 DEBUG [hconnection-0x25cd8e7f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:53:38,074 DEBUG [hconnection-0x24a6b8d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:53:38,079 DEBUG [hconnection-0x43d2b8e6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:53:38,079 DEBUG [hconnection-0x474938ac-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:53:38,080 DEBUG [hconnection-0x6903a248-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:53:38,083 DEBUG [hconnection-0x77cf76aa-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:53:38,083 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:60978, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:53:38,086 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:60994, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:53:38,086 DEBUG [hconnection-0x55a25988-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:53:38,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:38,088 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:32786, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:53:38,088 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:32782, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:53:38,088 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:32776, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:53:38,097 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:32818, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:53:38,100 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:32808, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:53:38,105 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:32792, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:53:38,106 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:32832, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:53:38,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:38,114 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:38,115 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:38,116 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:38,116 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:38,117 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:38,117 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:38,118 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:38,118 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:38,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:38,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:38,118 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:38,119 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:38,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:38,121 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:38,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:38,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:38,122 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:38,127 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:38,129 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:38,129 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:38,130 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:38,131 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:38,131 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:38,131 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:38,132 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:38,133 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fc3812e05598851193449f5684adf036 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-20 12:53:38,143 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=A 2023-04-20 12:53:38,144 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:53:38,144 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=B 2023-04-20 12:53:38,144 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:53:38,144 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=C 2023-04-20 12:53:38,144 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:53:38,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on fc3812e05598851193449f5684adf036 2023-04-20 12:53:38,186 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:38,306 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681995218187, max timestamp=1681995218236Num uniques 859; 2023-04-20 12:53:38,306 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681995218187, max timestamp=1681995218236Num uniques 859; 2023-04-20 12:53:38,312 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 12:53:38,312 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 12:53:38,332 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681995218187, max timestamp=1681995218267Num uniques 859; 2023-04-20 12:53:38,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 12:53:38,360 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=7 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/4380c2629118436596ef388bf4a5ec79 2023-04-20 12:53:38,471 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:38,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32786 deadline: 1681995278407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:38,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681995218377, max timestamp=1681995218408Num uniques 1018; 2023-04-20 12:53:38,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=6870, suffix heap size=133840,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=150 2023-04-20 12:53:38,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681995218377, max timestamp=1681995218408Num uniques 1018; 2023-04-20 12:53:38,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=6870, suffix heap size=133840,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=150 2023-04-20 12:53:38,479 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681995218377, max timestamp=1681995218411Num uniques 1018; 2023-04-20 12:53:38,479 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-20 12:53:38,547 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681995218514, max timestamp=1681995218526Num uniques 1027; 2023-04-20 12:53:38,547 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=6870, suffix heap size=134776,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=150 2023-04-20 12:53:38,560 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681995218514, max timestamp=1681995218526Num uniques 1009; 2023-04-20 12:53:38,560 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=46206, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-20 12:53:38,566 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681995218514, max timestamp=1681995218530Num uniques 1009; 2023-04-20 12:53:38,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-20 12:53:38,591 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=7 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/fd256aaf01fc45cf92a27550dee35a51 2023-04-20 12:53:38,617 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681995218601, max timestamp=1681995218612Num uniques 1009; 2023-04-20 12:53:38,617 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=10806, suffix heap size=132904,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=236 2023-04-20 12:53:38,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681995218601, max timestamp=1681995218612Num uniques 1009; 2023-04-20 12:53:38,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=10806, suffix heap size=132904,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=236 2023-04-20 12:53:38,628 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681995218601, max timestamp=1681995218609Num uniques 1009; 2023-04-20 12:53:38,628 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=46222, new segment data size=10806, suffix heap size=132904,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=236 2023-04-20 12:53:38,682 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=7 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/04f4bc85503742678ba998a28f08db27 2023-04-20 12:53:38,686 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13500, totalHeapSize=30920, min timestamp=1681995218640, max timestamp=1681995218681Num uniques 1095; 2023-04-20 12:53:38,686 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13500, totalHeapSize=30920, min timestamp=1681995218640, max timestamp=1681995218681Num uniques 1095; 2023-04-20 12:53:38,686 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=50140, new segment data size=13500, suffix heap size=141848,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=295 2023-04-20 12:53:38,686 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13500, totalHeapSize=30920, min timestamp=1681995218640, max timestamp=1681995218681Num uniques 1095; 2023-04-20 12:53:38,686 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=50140, new segment data size=13500, suffix heap size=141848,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=295 2023-04-20 12:53:38,686 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=50140, new segment data size=13500, suffix heap size=141848,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=295 2023-04-20 12:53:38,699 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/4380c2629118436596ef388bf4a5ec79 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/4380c2629118436596ef388bf4a5ec79 2023-04-20 12:53:38,714 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/4380c2629118436596ef388bf4a5ec79, entries=50, sequenceid=7, filesize=7.0 K 2023-04-20 12:53:38,717 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/fd256aaf01fc45cf92a27550dee35a51 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/fd256aaf01fc45cf92a27550dee35a51 2023-04-20 12:53:38,729 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/fd256aaf01fc45cf92a27550dee35a51, entries=50, sequenceid=7, filesize=7.0 K 2023-04-20 12:53:38,732 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/04f4bc85503742678ba998a28f08db27 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/04f4bc85503742678ba998a28f08db27 2023-04-20 12:53:38,744 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/04f4bc85503742678ba998a28f08db27, entries=50, sequenceid=7, filesize=7.0 K 2023-04-20 12:53:38,756 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=120.76 KB/123660 for fc3812e05598851193449f5684adf036 in 623ms, sequenceid=7, compaction requested=false 2023-04-20 12:53:38,759 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-20 12:53:38,763 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:38,763 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:38,764 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:38,765 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:38,765 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:38,765 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:38,766 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:38,771 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:38,771 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:38,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:38,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:38,771 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:38,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:38,771 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:38,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:38,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:38,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:38,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:38,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:38,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:38,774 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:38,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c12a790[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:38,774 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:38,775 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:38,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:38,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:38,777 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:38,778 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:38,778 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:38,778 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:38,778 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:38,779 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:38,780 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:38,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:38,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:38,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:38,780 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:38,780 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:38,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:38,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:38,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:38,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:38,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:38,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:38,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:38,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:38,790 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:38,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:38,790 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:38,790 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:38,790 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:38,790 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:38,795 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:38,795 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:38,795 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:38,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:38,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:38,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:38,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:38,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:38,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:38,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:38,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:38,797 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:38,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:38,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:38,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:38,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:38,800 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:38,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:38,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:38,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:38,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:38,815 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:38,815 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:38,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:38,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:38,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:38,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:38,816 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:38,816 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:38,816 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:38,816 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:38,816 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:38,816 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:38,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:38,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:38,816 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:38,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:38,817 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:38,820 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:38,820 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:38,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on fc3812e05598851193449f5684adf036 2023-04-20 12:53:38,824 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fc3812e05598851193449f5684adf036 3/3 column families, dataSize=147.60 KB heapSize=412.08 KB 2023-04-20 12:53:38,825 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=A 2023-04-20 12:53:38,825 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:53:38,825 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=B 2023-04-20 12:53:38,825 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:53:38,825 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=C 2023-04-20 12:53:38,826 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:53:38,848 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:38,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32782 deadline: 1681995278835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:38,850 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:38,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60994 deadline: 1681995278839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:38,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:38,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32786 deadline: 1681995278840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:38,858 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:38,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32792 deadline: 1681995278845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:38,860 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:38,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32832 deadline: 1681995278847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:38,886 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=112 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/178bcaefe7ff4cfdbe8f346159abe564 2023-04-20 12:53:38,921 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:38,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:38,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:38,957 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:38,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32782 deadline: 1681995278953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:38,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:38,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:38,958 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:38,958 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:38,958 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:38,959 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:38,959 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:38,960 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:38,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32786 deadline: 1681995278954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:38,961 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:38,961 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:38,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60994 deadline: 1681995278953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:38,962 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:38,963 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:38,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32792 deadline: 1681995278962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:38,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:38,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:38,963 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:38,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:38,964 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:38,964 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:38,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:38,964 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:38,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32832 deadline: 1681995278963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:38,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:38,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:38,965 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:38,970 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:38,970 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:38,970 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:38,970 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:38,971 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:38,972 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:38,972 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:38,972 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:38,972 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:38,972 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:38,972 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:38,973 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:38,973 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:38,973 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:38,973 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:38,973 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:38,980 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:38,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:38,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:38,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:38,980 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:38,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:38,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:38,982 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:38,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:38,982 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:38,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:38,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:38,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:38,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fa56233[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:38,983 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:38,983 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:38,988 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:38,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:38,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:38,988 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:38,989 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:38,989 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:38,989 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:38,989 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:38,995 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:38,995 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:38,995 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:38,995 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:38,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:38,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:38,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:38,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:38,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:38,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:38,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:38,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:39,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:39,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:39,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:39,002 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:39,002 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:39,002 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:39,002 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:39,002 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:39,006 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,006 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,006 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:39,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:39,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:39,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:39,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:39,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:39,007 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:39,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:39,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:39,011 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:39,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:39,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,020 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:39,020 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:39,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:39,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:39,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:39,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:39,025 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,025 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:39,025 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:39,025 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,025 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:39,026 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:39,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:39,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:39,025 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:39,029 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:39,029 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,029 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:39,029 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:39,129 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:39,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:39,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:39,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:39,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:39,146 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:39,146 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:39,163 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:39,163 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:39,164 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,164 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:39,165 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:39,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:39,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:39,166 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:39,167 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,167 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,167 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:39,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:39,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:39,168 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:39,169 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:39,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32786 deadline: 1681995279165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,170 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:39,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32782 deadline: 1681995279165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,171 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:39,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60994 deadline: 1681995279165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,172 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:39,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32792 deadline: 1681995279166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,173 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:39,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32832 deadline: 1681995279168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,175 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:39,175 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:39,176 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:39,176 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:39,176 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:39,182 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:39,182 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:39,182 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:39,182 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:39,182 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:39,182 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:39,186 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:39,186 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:39,187 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,187 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:39,187 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:39,189 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,189 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:39,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:39,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:39,190 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:39,190 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:39,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:39,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:39,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:39,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:39,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:39,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@445cf515[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:39,193 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:39,193 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:39,194 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:39,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:39,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:39,194 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:39,194 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:39,194 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:39,194 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,194 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:39,198 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:39,198 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:39,199 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:39,200 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:39,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:39,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:39,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:39,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:39,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:39,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:39,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:39,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:39,206 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:39,206 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:39,206 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:39,207 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:39,207 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:39,213 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,213 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:39,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:39,213 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:39,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:39,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:39,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:39,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:39,215 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:39,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:39,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:39,217 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:39,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,230 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:39,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:39,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:39,230 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,231 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:39,231 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:39,231 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,231 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:39,231 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:39,231 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:39,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:39,231 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:39,231 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:39,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:39,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:39,233 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:39,233 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:39,333 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:39,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:39,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:39,346 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=112 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/6087b0ef524e4d73a5434f8887e20acf 2023-04-20 12:53:39,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:39,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:39,350 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:39,350 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:39,359 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:39,359 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:39,359 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,360 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:39,360 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,360 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:39,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:39,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:39,361 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,361 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:39,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:39,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:39,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:39,363 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:39,367 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:39,367 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:39,367 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:39,367 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:39,370 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:39,381 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:39,381 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:39,382 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:39,382 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:39,382 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:39,389 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:39,397 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:39,397 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:39,398 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,398 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:39,398 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:39,403 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:39,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:39,403 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:39,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:39,409 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:39,409 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:39,409 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=112 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/4517f1c958094b5aae739a01a3af2e14 2023-04-20 12:53:39,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:39,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:39,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:39,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:39,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@337d8857[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:39,411 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:39,411 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:39,419 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:39,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:39,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:39,419 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:39,419 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:39,419 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:39,419 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,419 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:39,422 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,422 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:39,422 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:39,422 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:39,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:39,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:39,427 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/178bcaefe7ff4cfdbe8f346159abe564 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/178bcaefe7ff4cfdbe8f346159abe564 2023-04-20 12:53:39,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:39,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:39,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:39,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:39,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:39,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:39,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:39,432 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:39,432 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:39,432 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:39,432 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:39,432 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:39,434 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,434 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:39,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:39,434 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:39,435 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:39,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:39,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:39,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:39,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:39,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:39,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,438 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:39,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:39,439 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/178bcaefe7ff4cfdbe8f346159abe564, entries=150, sequenceid=112, filesize=11.6 K 2023-04-20 12:53:39,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,443 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/6087b0ef524e4d73a5434f8887e20acf as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/6087b0ef524e4d73a5434f8887e20acf 2023-04-20 12:53:39,446 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:39,446 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:39,446 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:39,446 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:39,447 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:39,447 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:39,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:39,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:39,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:39,447 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,447 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:39,447 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:39,447 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:39,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:39,448 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:39,448 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:39,455 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/6087b0ef524e4d73a5434f8887e20acf, entries=150, sequenceid=112, filesize=11.6 K 2023-04-20 12:53:39,462 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/4517f1c958094b5aae739a01a3af2e14 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/4517f1c958094b5aae739a01a3af2e14 2023-04-20 12:53:39,477 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:39,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32786 deadline: 1681995279473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,478 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:39,478 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/4517f1c958094b5aae739a01a3af2e14, entries=150, sequenceid=112, filesize=11.6 K 2023-04-20 12:53:39,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32782 deadline: 1681995279474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,479 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:39,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32792 deadline: 1681995279475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,484 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~412.03 KB/421920, currentSize=40.25 KB/41220 for fc3812e05598851193449f5684adf036 in 660ms, sequenceid=112, compaction requested=false 2023-04-20 12:53:39,485 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:39,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on fc3812e05598851193449f5684adf036 2023-04-20 12:53:39,486 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fc3812e05598851193449f5684adf036 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-20 12:53:39,487 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=A 2023-04-20 12:53:39,487 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:53:39,487 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=B 2023-04-20 12:53:39,487 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:53:39,487 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=C 2023-04-20 12:53:39,487 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:53:39,549 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:39,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:39,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:39,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:39,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:39,573 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:39,573 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:39,574 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:39,574 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:39,575 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,575 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:39,576 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,576 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:39,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:39,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:39,576 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,576 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:39,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:39,577 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:39,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:39,579 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:39,579 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:39,579 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:39,579 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:39,579 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:39,580 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:39,580 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:39,580 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:39,580 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:39,581 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:39,581 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:39,582 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:39,583 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:39,583 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:39,583 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,583 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:39,583 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:39,593 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,593 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:39,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:39,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:39,593 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:39,593 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:39,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:39,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:39,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:39,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:39,597 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681995219572, max timestamp=1681995219593Num uniques 859; 2023-04-20 12:53:39,598 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 12:53:39,598 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681995219572, max timestamp=1681995219595Num uniques 859; 2023-04-20 12:53:39,598 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 12:53:39,598 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681995219572, max timestamp=1681995219595Num uniques 859; 2023-04-20 12:53:39,598 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 12:53:39,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:39,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a1a878f[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:39,599 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:39,599 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:39,601 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:39,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:39,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:39,602 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:39,602 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,602 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:39,602 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:39,602 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:39,603 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:39,603 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:39,603 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:39,605 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:39,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:39,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:39,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:39,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:39,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:39,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:39,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:39,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:39,609 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:39,609 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:39,609 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:39,609 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:39,609 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:39,618 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:39,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:39,619 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,619 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:39,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:39,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:39,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:39,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:39,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,620 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:39,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:39,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:39,622 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:39,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,628 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:39,628 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,628 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:39,628 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:39,629 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:39,629 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:39,629 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:39,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:39,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:39,629 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:39,629 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:39,629 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,629 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:39,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:39,629 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:39,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:39,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:39,651 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:39,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32832 deadline: 1681995279649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,654 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:39,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60994 deadline: 1681995279651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,730 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:39,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:39,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:39,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:39,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:39,738 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:39,739 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:39,739 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:39,740 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:39,740 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,740 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:39,741 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:39,741 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:39,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:39,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:39,742 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,742 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:39,742 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:39,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:39,743 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:39,743 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:39,744 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:39,744 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:39,744 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:39,746 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:39,747 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:39,747 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:39,747 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:39,747 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:39,747 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:39,747 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:39,748 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:39,748 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:39,748 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,748 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:39,748 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:39,758 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:39,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32832 deadline: 1681995279757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,759 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:39,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60994 deadline: 1681995279757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,759 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:39,759 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:39,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:39,760 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:39,760 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:39,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:39,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:39,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:39,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:39,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:39,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e6ac605[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:39,764 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:39,764 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:39,768 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:39,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:39,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:39,768 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:39,768 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:39,768 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:39,768 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,768 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:39,769 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,769 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:39,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:39,769 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:39,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:39,769 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:39,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:39,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:39,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:39,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:39,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:39,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:39,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:39,778 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:39,778 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:39,778 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:39,778 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:39,778 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:39,784 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,785 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:39,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,785 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:39,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:39,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:39,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:39,786 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:39,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:39,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:39,788 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:39,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:39,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:39,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,797 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:39,797 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:39,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:39,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:39,798 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:39,798 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:39,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:39,801 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:39,801 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:39,800 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:39,801 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:39,801 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,801 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:39,801 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:39,801 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:39,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:39,802 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:39,901 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:39,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:39,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:39,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:39,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:39,908 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:39,908 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:39,909 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:39,909 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:39,910 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,910 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:39,910 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:39,910 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:39,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:39,911 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,911 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:39,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:39,911 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:39,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:39,912 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:39,912 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:39,912 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:39,912 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:39,912 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:39,913 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:39,913 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:39,913 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:39,914 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:39,914 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:39,914 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:39,914 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:39,916 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:39,916 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:39,916 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,916 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:39,916 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:39,921 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:39,921 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:39,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:39,922 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:39,922 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:39,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:39,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:39,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:39,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:39,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:39,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d3d7880[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:39,924 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:39,924 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:39,926 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:39,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:39,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:39,926 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:39,926 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:39,926 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,926 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:39,926 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:39,927 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,927 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:39,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:39,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:39,928 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:39,928 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:39,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:39,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:39,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:39,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:39,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:39,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:39,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:39,932 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:39,932 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:39,932 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:39,932 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:39,932 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:39,933 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,933 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:39,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,933 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:39,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:39,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:39,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:39,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:39,934 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:39,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:39,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:39,937 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=122 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/29269cc720484c778e35bcbdcb14c9b1 2023-04-20 12:53:39,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,939 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:39,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:39,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,946 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,946 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:39,946 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:39,947 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,947 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:39,947 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:39,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:39,947 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:39,947 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:39,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:39,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:39,948 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:39,948 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:39,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:39,948 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:39,948 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:39,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:39,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:39,961 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:39,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32832 deadline: 1681995279961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,963 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:39,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60994 deadline: 1681995279962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,978 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=122 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/f856402354ad47a9ae6b6db58bbcd048 2023-04-20 12:53:39,982 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:39,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32786 deadline: 1681995279982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,983 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:39,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32792 deadline: 1681995279982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:39,984 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:39,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32782 deadline: 1681995279983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,048 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:40,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:40,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:40,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:40,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:40,054 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:40,054 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:40,059 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:40,059 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:40,060 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:40,060 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:40,061 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:40,061 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:40,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:40,061 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,061 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:40,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:40,062 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:40,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:40,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:40,062 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:40,063 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:40,063 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:40,065 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:40,065 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:40,066 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:40,067 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:40,067 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:40,067 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:40,067 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:40,067 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:40,067 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:40,067 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:40,067 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:40,067 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,067 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:40,067 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:40,069 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,069 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:40,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:40,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:40,069 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:40,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:40,069 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:40,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:40,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:40,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:40,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:40,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42dd7440[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:40,071 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:40,071 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:40,072 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:40,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:40,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:40,072 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,072 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:40,072 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:40,072 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:40,072 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:40,073 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:40,073 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:40,073 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:40,073 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:40,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:40,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:40,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:40,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:40,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:40,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:40,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:40,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:40,082 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:40,082 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:40,082 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:40,082 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:40,082 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:40,088 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:40,088 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:40,088 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:40,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:40,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:40,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:40,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:40,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:40,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:40,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:40,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:40,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:40,090 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:40,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:40,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:40,093 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:40,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:40,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,103 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:40,103 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:40,103 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:40,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:40,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:40,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:40,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:40,103 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,107 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:40,107 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:40,107 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,107 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:40,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:40,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:40,107 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:40,107 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:40,107 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:40,107 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:40,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:40,207 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:40,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:40,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:40,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:40,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:40,224 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:40,224 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:40,225 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:40,225 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:40,226 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:40,226 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:40,227 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:40,227 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:40,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:40,227 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,227 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:40,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:40,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:40,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:40,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:40,229 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:40,229 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:40,229 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:40,234 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:40,235 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:40,238 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:40,239 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:40,239 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:40,239 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:40,239 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:40,240 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:40,240 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:40,240 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:40,240 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:40,240 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,240 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:40,240 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:40,247 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:40,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:40,247 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:40,247 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:40,247 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:40,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:40,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:40,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:40,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:40,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:40,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c55b56b[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:40,251 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:40,251 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:40,255 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:40,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:40,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:40,255 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:40,255 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:40,255 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,255 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:40,255 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:40,256 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,256 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:40,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:40,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:40,256 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:40,256 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:40,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:40,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:40,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:40,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:40,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:40,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:40,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:40,261 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:40,261 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:40,261 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:40,261 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:40,261 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:40,268 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:40,268 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:40,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:40,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:40,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:40,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:40,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:40,269 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:40,269 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:40,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:40,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:40,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:40,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32832 deadline: 1681995280266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:40,270 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:40,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:40,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:40,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,271 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,272 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:40,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60994 deadline: 1681995280268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:40,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:40,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,276 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:40,276 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:40,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:40,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:40,276 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,277 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:40,277 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:40,277 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,277 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:40,277 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:40,277 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:40,277 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:40,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:40,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:40,279 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:40,279 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:40,279 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:40,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:40,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:40,379 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:40,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:40,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:40,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:40,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:40,385 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:40,386 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:40,388 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:40,388 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:40,389 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:40,389 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:40,390 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:40,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:40,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:40,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:40,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:40,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:40,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:40,392 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:40,392 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,392 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:40,392 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:40,392 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:40,392 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:40,396 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:40,396 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,396 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:40,396 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:40,396 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:40,397 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:40,397 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:40,397 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:40,397 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:40,397 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:40,397 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:40,397 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,397 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:40,397 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:40,409 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,409 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:40,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:40,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:40,410 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:40,410 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:40,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:40,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:40,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:40,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:40,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:40,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e70c0a[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:40,412 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:40,412 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:40,413 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:40,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:40,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:40,413 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:40,413 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:40,413 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:40,413 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,413 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:40,415 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:40,415 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:40,415 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:40,419 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:40,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:40,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:40,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:40,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:40,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:40,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:40,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,422 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=122 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/57ef34ba05054f47b630839ed7c7b41e 2023-04-20 12:53:40,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:40,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:40,423 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:40,423 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:40,423 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:40,423 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:40,423 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:40,430 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:40,430 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:40,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:40,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:40,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:40,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:40,431 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:40,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:40,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:40,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:40,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:40,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:40,432 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:40,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:40,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:40,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,434 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:40,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:40,436 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/29269cc720484c778e35bcbdcb14c9b1 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/29269cc720484c778e35bcbdcb14c9b1 2023-04-20 12:53:40,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,439 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:40,439 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,439 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:40,439 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:40,439 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,439 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:40,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:40,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:40,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:40,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:40,439 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:40,439 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:40,440 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:40,440 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:40,440 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:40,440 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:40,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:40,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:40,448 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/29269cc720484c778e35bcbdcb14c9b1, entries=250, sequenceid=122, filesize=16.2 K 2023-04-20 12:53:40,450 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/f856402354ad47a9ae6b6db58bbcd048 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/f856402354ad47a9ae6b6db58bbcd048 2023-04-20 12:53:40,464 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/f856402354ad47a9ae6b6db58bbcd048, entries=150, sequenceid=122, filesize=11.6 K 2023-04-20 12:53:40,469 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/57ef34ba05054f47b630839ed7c7b41e as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/57ef34ba05054f47b630839ed7c7b41e 2023-04-20 12:53:40,477 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/57ef34ba05054f47b630839ed7c7b41e, entries=150, sequenceid=122, filesize=11.6 K 2023-04-20 12:53:40,479 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for fc3812e05598851193449f5684adf036 in 993ms, sequenceid=122, compaction requested=true 2023-04-20 12:53:40,480 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:40,482 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:53:40,483 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:53:40,483 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:53:40,483 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:53:40,484 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:53:40,487 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 30972 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:53:40,490 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): fc3812e05598851193449f5684adf036/B is initiating minor compaction (all files) 2023-04-20 12:53:40,490 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fc3812e05598851193449f5684adf036/B in TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:40,490 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/fd256aaf01fc45cf92a27550dee35a51, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/6087b0ef524e4d73a5434f8887e20acf, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/f856402354ad47a9ae6b6db58bbcd048] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp, totalSize=30.2 K 2023-04-20 12:53:40,492 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting fd256aaf01fc45cf92a27550dee35a51, keycount=50, bloomtype=ROW, size=7.0 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1681995218112 2023-04-20 12:53:40,494 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 6087b0ef524e4d73a5434f8887e20acf, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1681995218709 2023-04-20 12:53:40,494 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting f856402354ad47a9ae6b6db58bbcd048, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=122, earliestPutTs=1681995218840 2023-04-20 12:53:40,502 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35652 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:53:40,503 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): fc3812e05598851193449f5684adf036/A is initiating minor compaction (all files) 2023-04-20 12:53:40,503 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fc3812e05598851193449f5684adf036/A in TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:40,503 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/4380c2629118436596ef388bf4a5ec79, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/178bcaefe7ff4cfdbe8f346159abe564, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/29269cc720484c778e35bcbdcb14c9b1] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp, totalSize=34.8 K 2023-04-20 12:53:40,504 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 4380c2629118436596ef388bf4a5ec79, keycount=50, bloomtype=ROW, size=7.0 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1681995218112 2023-04-20 12:53:40,505 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 178bcaefe7ff4cfdbe8f346159abe564, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1681995218709 2023-04-20 12:53:40,506 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 29269cc720484c778e35bcbdcb14c9b1, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=122, earliestPutTs=1681995218827 2023-04-20 12:53:40,534 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): fc3812e05598851193449f5684adf036#A#compaction#9 average throughput is 1.09 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:53:40,540 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:40,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:40,549 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): fc3812e05598851193449f5684adf036#B#compaction#10 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:53:40,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:40,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:40,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:40,564 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:40,564 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:40,567 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:40,567 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:40,578 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:40,578 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:40,579 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:40,579 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:40,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:40,580 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,580 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:40,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:40,580 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:40,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:40,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:40,581 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:40,587 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:40,587 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:40,587 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:40,587 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:40,588 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:40,594 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:40,599 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:40,602 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:40,602 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:40,603 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fc3812e05598851193449f5684adf036 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-20 12:53:40,603 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=A 2023-04-20 12:53:40,603 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:53:40,603 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=B 2023-04-20 12:53:40,603 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:53:40,603 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=C 2023-04-20 12:53:40,604 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:53:40,659 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/dc2f827330d94cd9a578332dd10a5e73 2023-04-20 12:53:40,701 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/855f3a218b3f457a9578927285d93556 2023-04-20 12:53:40,750 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/4d34e0a359ac4248804192649298ce59 2023-04-20 12:53:40,765 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/dc2f827330d94cd9a578332dd10a5e73 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/dc2f827330d94cd9a578332dd10a5e73 2023-04-20 12:53:40,780 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/dc2f827330d94cd9a578332dd10a5e73, entries=150, sequenceid=158, filesize=11.8 K 2023-04-20 12:53:40,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on fc3812e05598851193449f5684adf036 2023-04-20 12:53:40,785 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:40,791 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/855f3a218b3f457a9578927285d93556 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/855f3a218b3f457a9578927285d93556 2023-04-20 12:53:40,823 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/855f3a218b3f457a9578927285d93556, entries=150, sequenceid=158, filesize=11.8 K 2023-04-20 12:53:40,830 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/4d34e0a359ac4248804192649298ce59 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/4d34e0a359ac4248804192649298ce59 2023-04-20 12:53:40,843 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/4d34e0a359ac4248804192649298ce59, entries=150, sequenceid=158, filesize=11.8 K 2023-04-20 12:53:40,844 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=40.25 KB/41220 for fc3812e05598851193449f5684adf036 in 242ms, sequenceid=158, compaction requested=true 2023-04-20 12:53:40,845 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:40,845 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:40,845 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:40,847 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:40,847 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,847 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:40,847 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:40,848 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,848 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:40,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:40,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:40,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:40,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:40,849 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:40,849 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:40,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:40,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on fc3812e05598851193449f5684adf036 2023-04-20 12:53:40,852 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fc3812e05598851193449f5684adf036 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-20 12:53:40,852 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=A 2023-04-20 12:53:40,852 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:53:40,852 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=B 2023-04-20 12:53:40,852 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:53:40,852 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=C 2023-04-20 12:53:40,853 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:53:40,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:40,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:40,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72e89d6e[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:40,854 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:40,855 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:40,856 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:40,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:40,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:40,856 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:40,857 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:40,857 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:40,858 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,858 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:40,859 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:40,859 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,859 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:40,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:40,859 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:40,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:40,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:40,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:40,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:40,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:40,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:40,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:40,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:40,865 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:40,865 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:40,865 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:40,865 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:40,865 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:40,877 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:40,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:40,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:40,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:40,879 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:40,879 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:40,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:40,879 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:40,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:40,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:40,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:40,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:40,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:40,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:40,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:40,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,881 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:40,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:40,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,885 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:40,885 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:40,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:40,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:40,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:40,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:40,886 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,886 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:40,886 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:40,886 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:40,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:40,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:40,886 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:40,886 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:40,886 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:40,886 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:40,887 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:40,887 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:40,900 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/767c75e9bff84f358366527ace38efb8 2023-04-20 12:53:40,981 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681995220934, max timestamp=1681995220977Num uniques 859; 2023-04-20 12:53:40,981 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 12:53:40,981 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681995220934, max timestamp=1681995220977Num uniques 859; 2023-04-20 12:53:40,981 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 12:53:40,981 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681995220934, max timestamp=1681995220977Num uniques 859; 2023-04-20 12:53:40,981 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 12:53:40,987 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:40,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:40,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:40,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:40,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:40,994 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:40,994 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:40,998 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:40,998 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:40,999 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:40,999 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:41,001 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:41,001 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:41,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:41,001 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,001 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:41,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:41,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:41,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:41,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:41,002 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:41,002 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:41,003 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:41,003 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:41,003 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:41,004 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:41,005 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:41,005 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:41,005 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:41,005 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:41,005 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:41,005 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:41,005 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:41,005 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:41,005 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,005 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:41,005 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:41,013 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:41,013 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:41,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:41,014 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:41,014 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:41,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:41,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:41,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:41,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:41,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:41,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bbe9807[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:41,016 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:41,016 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:41,017 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:41,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:41,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:41,017 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:41,017 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:41,018 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:41,018 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,018 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:41,019 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,019 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:41,019 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:41,019 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:41,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:41,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:41,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:41,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:41,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:41,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:41,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:41,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:41,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:41,027 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:41,027 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:41,027 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:41,027 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:41,027 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:41,034 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:41,035 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:41,035 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:41,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:41,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:41,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:41,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:41,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:41,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:41,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:41,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:41,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:41,041 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:41,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60994 deadline: 1681995281030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,042 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:41,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32782 deadline: 1681995281031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,045 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:41,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:41,049 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:41,049 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32792 deadline: 1681995281036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:41,050 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:41,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32786 deadline: 1681995281036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,050 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/114a10a003ef498880821855ded8127c as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/114a10a003ef498880821855ded8127c 2023-04-20 12:53:41,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:41,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:41,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,053 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/43ef446734d849da8b24680e7f86d638 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/43ef446734d849da8b24680e7f86d638 2023-04-20 12:53:41,053 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:41,053 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,056 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:41,056 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:41,056 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,056 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:41,056 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:41,056 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:41,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:41,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:41,055 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:41,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:41,055 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:41,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:41,061 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:41,061 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:41,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:41,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:41,062 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:41,065 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:41,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32832 deadline: 1681995281043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,084 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fc3812e05598851193449f5684adf036/B of fc3812e05598851193449f5684adf036 into 43ef446734d849da8b24680e7f86d638(size=11.7 K), total size for store is 23.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:53:41,084 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:41,084 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fc3812e05598851193449f5684adf036/A of fc3812e05598851193449f5684adf036 into 114a10a003ef498880821855ded8127c(size=11.7 K), total size for store is 23.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:53:41,085 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:41,085 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036., storeName=fc3812e05598851193449f5684adf036/B, priority=13, startTime=1681995220483; duration=0sec 2023-04-20 12:53:41,085 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036., storeName=fc3812e05598851193449f5684adf036/A, priority=13, startTime=1681995220482; duration=0sec 2023-04-20 12:53:41,086 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:53:41,086 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:53:41,086 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 12:53:41,089 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 43008 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 12:53:41,089 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): fc3812e05598851193449f5684adf036/C is initiating minor compaction (all files) 2023-04-20 12:53:41,089 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fc3812e05598851193449f5684adf036/C in TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:41,090 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/04f4bc85503742678ba998a28f08db27, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/4517f1c958094b5aae739a01a3af2e14, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/57ef34ba05054f47b630839ed7c7b41e, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/4d34e0a359ac4248804192649298ce59] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp, totalSize=42 K 2023-04-20 12:53:41,090 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 04f4bc85503742678ba998a28f08db27, keycount=50, bloomtype=ROW, size=7.0 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1681995218112 2023-04-20 12:53:41,093 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 4517f1c958094b5aae739a01a3af2e14, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1681995218709 2023-04-20 12:53:41,095 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 57ef34ba05054f47b630839ed7c7b41e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=122, earliestPutTs=1681995218840 2023-04-20 12:53:41,096 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 4d34e0a359ac4248804192649298ce59, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1681995219641 2023-04-20 12:53:41,125 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): fc3812e05598851193449f5684adf036#C#compaction#16 average throughput is 0.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:53:41,150 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:41,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60994 deadline: 1681995281144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,152 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:41,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32782 deadline: 1681995281145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,155 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:41,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32792 deadline: 1681995281154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,161 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:41,162 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:41,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32786 deadline: 1681995281159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:41,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:41,169 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:41,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:41,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32832 deadline: 1681995281167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:41,173 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:41,173 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:41,174 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:41,174 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:41,174 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:41,174 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:41,175 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,175 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:41,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:41,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:41,176 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:41,176 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:41,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:41,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:41,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:41,189 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:41,190 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:41,190 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:41,190 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:41,190 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:41,191 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:41,191 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:41,191 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:41,191 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:41,191 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:41,192 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:41,192 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:41,192 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:41,192 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:41,192 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,192 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:41,192 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:41,193 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:41,193 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:41,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:41,194 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:41,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:41,194 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:41,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:41,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:41,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:41,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:41,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a8180f6[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:41,195 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:41,195 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:41,197 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:41,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:41,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:41,197 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:41,197 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:41,197 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,197 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:41,197 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:41,198 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,198 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:41,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:41,198 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:41,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:41,198 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:41,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:41,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:41,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:41,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:41,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:41,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:41,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:41,202 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:41,202 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:41,202 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:41,202 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:41,202 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:41,204 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:41,205 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:41,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:41,205 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:41,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:41,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:41,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:41,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:41,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:41,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:41,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:41,205 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:41,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:41,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:41,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:41,207 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:41,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:41,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,217 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,217 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:41,218 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:41,218 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:41,218 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:41,218 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:41,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:41,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:41,218 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:41,219 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,219 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:41,219 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:41,219 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:41,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:41,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:41,219 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:41,220 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:41,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:41,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:41,320 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:41,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:41,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:41,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:41,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:41,326 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:41,326 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:41,328 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:41,328 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:41,328 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:41,329 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:41,330 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,330 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:41,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:41,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:41,331 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,331 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:41,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:41,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:41,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:41,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:41,332 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:41,332 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:41,332 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:41,332 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:41,332 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:41,333 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:41,334 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:41,334 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:41,334 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:41,334 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:41,334 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:41,334 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:41,334 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:41,334 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:41,335 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,335 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:41,335 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:41,337 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-20 12:53:41,343 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,343 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:41,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:41,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:41,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:41,344 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:41,344 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:41,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:41,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:41,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:41,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:41,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26ba48f0[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:41,345 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:41,345 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:41,347 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:41,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:41,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:41,347 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:41,347 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:41,347 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:41,348 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,348 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:41,349 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:41,349 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,349 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:41,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:41,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:41,349 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:41,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:41,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:41,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:41,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:41,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:41,353 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/9e910ba6e49f4ad5ad677c7837d4902a 2023-04-20 12:53:41,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:41,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:41,355 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:41,355 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:41,355 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:41,356 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:41,356 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:41,360 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:41,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32782 deadline: 1681995281357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,363 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:41,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:41,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:41,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:41,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:41,364 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,364 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:41,364 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:41,365 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:41,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32792 deadline: 1681995281358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:41,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:41,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:41,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:41,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:41,365 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:41,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:41,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:41,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:41,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60994 deadline: 1681995281363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,373 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:41,378 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:41,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32786 deadline: 1681995281368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:41,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,385 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:41,385 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:41,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:41,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:41,385 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:41,385 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,385 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:41,386 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:41,386 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,386 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:41,386 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:41,386 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:41,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:41,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:41,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:41,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:41,387 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:41,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32832 deadline: 1681995281384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,388 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:41,388 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:41,437 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/4d56472ab5dc43c7a7945e2605ec59ca 2023-04-20 12:53:41,453 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/767c75e9bff84f358366527ace38efb8 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/767c75e9bff84f358366527ace38efb8 2023-04-20 12:53:41,464 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-04-20 12:53:41,465 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-04-20 12:53:41,468 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/767c75e9bff84f358366527ace38efb8, entries=350, sequenceid=168, filesize=21.1 K 2023-04-20 12:53:41,470 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/9e910ba6e49f4ad5ad677c7837d4902a as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/9e910ba6e49f4ad5ad677c7837d4902a 2023-04-20 12:53:41,482 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/9e910ba6e49f4ad5ad677c7837d4902a, entries=150, sequenceid=168, filesize=11.8 K 2023-04-20 12:53:41,495 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:41,497 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/4d56472ab5dc43c7a7945e2605ec59ca as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/4d56472ab5dc43c7a7945e2605ec59ca 2023-04-20 12:53:41,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:41,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:41,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:41,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:41,509 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:41,509 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:41,510 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:41,510 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:41,511 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:41,511 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:41,512 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,512 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:41,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:41,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:41,512 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,512 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:41,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:41,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:41,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:41,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:41,513 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:41,513 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/4d56472ab5dc43c7a7945e2605ec59ca, entries=150, sequenceid=168, filesize=11.8 K 2023-04-20 12:53:41,514 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:41,514 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:41,514 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:41,515 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:41,516 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:41,516 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for fc3812e05598851193449f5684adf036 in 664ms, sequenceid=168, compaction requested=true 2023-04-20 12:53:41,516 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:41,517 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:53:41,517 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:53:41,517 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:53:41,517 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:41,517 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-20 12:53:41,517 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:41,517 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:41,520 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:41,520 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fc3812e05598851193449f5684adf036 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-20 12:53:41,520 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=A 2023-04-20 12:53:41,520 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:53:41,520 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=B 2023-04-20 12:53:41,520 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:53:41,520 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=C 2023-04-20 12:53:41,520 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:53:41,524 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 45625 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:53:41,524 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): fc3812e05598851193449f5684adf036/A is initiating minor compaction (all files) 2023-04-20 12:53:41,524 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fc3812e05598851193449f5684adf036/A in TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:41,525 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/114a10a003ef498880821855ded8127c, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/dc2f827330d94cd9a578332dd10a5e73, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/767c75e9bff84f358366527ace38efb8] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp, totalSize=44.6 K 2023-04-20 12:53:41,528 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 114a10a003ef498880821855ded8127c, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=122, earliestPutTs=1681995218840 2023-04-20 12:53:41,528 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting dc2f827330d94cd9a578332dd10a5e73, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1681995219641 2023-04-20 12:53:41,529 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 767c75e9bff84f358366527ace38efb8, keycount=350, bloomtype=ROW, size=21.1 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1681995220780 2023-04-20 12:53:41,558 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): fc3812e05598851193449f5684adf036#A#compaction#19 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:53:41,572 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=207 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/b169c2caecc4494bafed10149a40d74d 2023-04-20 12:53:41,600 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/0b26314a595f497cad9b2f78ecc05890 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/0b26314a595f497cad9b2f78ecc05890 2023-04-20 12:53:41,607 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/15bd28d10e954c4d96a60c94428f59d5 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/15bd28d10e954c4d96a60c94428f59d5 2023-04-20 12:53:41,616 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in fc3812e05598851193449f5684adf036/C of fc3812e05598851193449f5684adf036 into 0b26314a595f497cad9b2f78ecc05890(size=11.9 K), total size for store is 23.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:53:41,618 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:41,618 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036., storeName=fc3812e05598851193449f5684adf036/C, priority=12, startTime=1681995220483; duration=0sec 2023-04-20 12:53:41,618 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-20 12:53:41,619 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:53:41,621 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fc3812e05598851193449f5684adf036/A of fc3812e05598851193449f5684adf036 into 15bd28d10e954c4d96a60c94428f59d5(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:53:41,636 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36061 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:53:41,637 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:41,637 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): fc3812e05598851193449f5684adf036/B is initiating minor compaction (all files) 2023-04-20 12:53:41,637 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036., storeName=fc3812e05598851193449f5684adf036/A, priority=13, startTime=1681995221517; duration=0sec 2023-04-20 12:53:41,637 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fc3812e05598851193449f5684adf036/B in TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:41,638 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:53:41,638 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/43ef446734d849da8b24680e7f86d638, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/855f3a218b3f457a9578927285d93556, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/9e910ba6e49f4ad5ad677c7837d4902a] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp, totalSize=35.2 K 2023-04-20 12:53:41,638 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-20 12:53:41,640 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 43ef446734d849da8b24680e7f86d638, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=122, earliestPutTs=1681995218840 2023-04-20 12:53:41,641 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 855f3a218b3f457a9578927285d93556, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1681995219641 2023-04-20 12:53:41,642 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-20 12:53:41,642 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-20 12:53:41,642 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. because compaction request was cancelled 2023-04-20 12:53:41,642 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 9e910ba6e49f4ad5ad677c7837d4902a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1681995220780 2023-04-20 12:53:41,671 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): fc3812e05598851193449f5684adf036#B#compaction#21 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:53:41,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on fc3812e05598851193449f5684adf036 2023-04-20 12:53:41,673 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:41,691 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:41,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60994 deadline: 1681995281687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,696 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:41,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32782 deadline: 1681995281690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,699 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:41,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32832 deadline: 1681995281691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,699 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:41,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32792 deadline: 1681995281691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,700 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:41,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32786 deadline: 1681995281692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,794 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:41,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60994 deadline: 1681995281793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,802 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:41,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32782 deadline: 1681995281800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,802 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:41,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32786 deadline: 1681995281802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,803 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:41,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32792 deadline: 1681995281802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:41,998 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:41,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60994 deadline: 1681995281997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:42,005 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:42,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32782 deadline: 1681995282003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:42,006 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:42,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32786 deadline: 1681995282005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:42,007 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:42,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32792 deadline: 1681995282005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:42,055 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=207 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/b2527240d2ca4f2f8d735a88d4aa7835 2023-04-20 12:53:42,114 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/d303006629324fbcad8537dd27e10df4 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/d303006629324fbcad8537dd27e10df4 2023-04-20 12:53:42,123 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fc3812e05598851193449f5684adf036/B of fc3812e05598851193449f5684adf036 into d303006629324fbcad8537dd27e10df4(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:53:42,123 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:42,124 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036., storeName=fc3812e05598851193449f5684adf036/B, priority=13, startTime=1681995221517; duration=0sec 2023-04-20 12:53:42,125 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:53:42,203 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:42,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32832 deadline: 1681995282201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:42,301 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:42,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60994 deadline: 1681995282300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:42,311 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:42,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32786 deadline: 1681995282308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:42,312 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:42,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32792 deadline: 1681995282308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:42,312 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:42,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32782 deadline: 1681995282309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:42,484 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=207 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/523c7f28b3bd476e8d3bf0dcfa8ba35f 2023-04-20 12:53:42,492 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/b169c2caecc4494bafed10149a40d74d as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/b169c2caecc4494bafed10149a40d74d 2023-04-20 12:53:42,502 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/b169c2caecc4494bafed10149a40d74d, entries=150, sequenceid=207, filesize=11.8 K 2023-04-20 12:53:42,505 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/b2527240d2ca4f2f8d735a88d4aa7835 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/b2527240d2ca4f2f8d735a88d4aa7835 2023-04-20 12:53:42,514 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/b2527240d2ca4f2f8d735a88d4aa7835, entries=150, sequenceid=207, filesize=11.8 K 2023-04-20 12:53:42,520 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/523c7f28b3bd476e8d3bf0dcfa8ba35f as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/523c7f28b3bd476e8d3bf0dcfa8ba35f 2023-04-20 12:53:42,534 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/523c7f28b3bd476e8d3bf0dcfa8ba35f, entries=150, sequenceid=207, filesize=11.8 K 2023-04-20 12:53:42,536 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=53.67 KB/54960 for fc3812e05598851193449f5684adf036 in 1016ms, sequenceid=207, compaction requested=true 2023-04-20 12:53:42,536 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:42,536 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:42,536 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:42,537 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:42,537 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:42,537 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:42,537 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:42,539 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:42,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:42,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:42,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:42,539 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:42,540 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:42,540 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:42,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:42,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:42,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:42,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:42,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:42,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:42,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@108fa2a4[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:42,543 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:42,543 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:42,546 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:42,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:42,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:42,546 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:42,546 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:42,546 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:42,547 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:42,547 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:42,547 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:42,547 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:42,547 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:42,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:42,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:42,547 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:42,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:42,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:42,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:42,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:42,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:42,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:42,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:42,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:42,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:42,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:42,551 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:42,551 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:42,551 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:42,551 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:42,551 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:42,560 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:42,560 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:42,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:42,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:42,560 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:42,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:42,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:42,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:42,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:42,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:42,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:42,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:42,562 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:42,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:42,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:42,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:42,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:42,564 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:42,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:42,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:42,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:42,568 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:42,568 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:42,568 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:42,568 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:42,568 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:42,568 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:42,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:42,568 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:42,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:42,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:42,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:42,569 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:42,570 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:42,570 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:42,570 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:42,570 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:42,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:42,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:42,670 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:42,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:42,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:42,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:42,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:42,674 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:42,674 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:42,675 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:42,675 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:42,676 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:42,676 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:42,677 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:42,677 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:42,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:42,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:42,677 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:42,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:42,677 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:42,677 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:42,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:42,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:42,678 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:42,678 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:42,678 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:42,678 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:42,678 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:42,679 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:42,682 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:42,682 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:42,682 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:42,682 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:42,682 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fc3812e05598851193449f5684adf036 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-20 12:53:42,683 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=A 2023-04-20 12:53:42,683 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:53:42,683 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=B 2023-04-20 12:53:42,683 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:53:42,683 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=C 2023-04-20 12:53:42,683 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:53:42,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on fc3812e05598851193449f5684adf036 2023-04-20 12:53:42,810 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:42,839 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:42,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32792 deadline: 1681995282834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:42,840 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681995222828, max timestamp=1681995222837Num uniques 859; 2023-04-20 12:53:42,840 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 12:53:42,840 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:42,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32786 deadline: 1681995282835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:42,840 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681995222828, max timestamp=1681995222837Num uniques 859; 2023-04-20 12:53:42,840 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 12:53:42,843 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681995222828, max timestamp=1681995222837Num uniques 859; 2023-04-20 12:53:42,843 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 12:53:42,892 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:42,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60994 deadline: 1681995282886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:42,893 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:42,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32782 deadline: 1681995282887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:42,942 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:42,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32792 deadline: 1681995282941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:42,946 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:42,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32786 deadline: 1681995282944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:42,996 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:42,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60994 deadline: 1681995282994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:42,998 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:42,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32782 deadline: 1681995282995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:43,117 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=221 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/d2510056be1846ef912b783cbf62405e 2023-04-20 12:53:43,152 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:43,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32792 deadline: 1681995283149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:43,153 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:43,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32786 deadline: 1681995283151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:43,156 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-20 12:53:43,157 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-04-20 12:53:43,199 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:43,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60994 deadline: 1681995283197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:43,202 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:43,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32782 deadline: 1681995283200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:43,215 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:43,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32832 deadline: 1681995283210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:43,457 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:43,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32792 deadline: 1681995283454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:43,457 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:43,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32786 deadline: 1681995283455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:43,504 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:43,504 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:43,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60994 deadline: 1681995283502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:43,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32782 deadline: 1681995283503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:43,557 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=221 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/51f1b23a2d8f4dde9c1879ea1c8bf37f 2023-04-20 12:53:43,963 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:43,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32786 deadline: 1681995283961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:43,968 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:43,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32792 deadline: 1681995283967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:43,990 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=221 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/254fbb5994d74ac1bd4886aa60a4f25c 2023-04-20 12:53:43,996 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/d2510056be1846ef912b783cbf62405e as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/d2510056be1846ef912b783cbf62405e 2023-04-20 12:53:44,003 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/d2510056be1846ef912b783cbf62405e, entries=150, sequenceid=221, filesize=11.8 K 2023-04-20 12:53:44,004 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/51f1b23a2d8f4dde9c1879ea1c8bf37f as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/51f1b23a2d8f4dde9c1879ea1c8bf37f 2023-04-20 12:53:44,012 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:44,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60994 deadline: 1681995284008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,013 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:44,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32782 deadline: 1681995284013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,015 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/51f1b23a2d8f4dde9c1879ea1c8bf37f, entries=150, sequenceid=221, filesize=11.8 K 2023-04-20 12:53:44,018 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/254fbb5994d74ac1bd4886aa60a4f25c as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/254fbb5994d74ac1bd4886aa60a4f25c 2023-04-20 12:53:44,042 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/254fbb5994d74ac1bd4886aa60a4f25c, entries=150, sequenceid=221, filesize=11.8 K 2023-04-20 12:53:44,044 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for fc3812e05598851193449f5684adf036 in 1362ms, sequenceid=221, compaction requested=true 2023-04-20 12:53:44,044 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:44,044 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:44,044 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:44,045 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:44,045 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,045 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:44,045 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:44,046 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:44,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:44,047 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:44,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:44,047 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:44,047 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:44,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:44,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:44,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:44,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:44,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f5e7e12[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:44,049 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:44,049 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:44,050 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:44,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:44,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:44,050 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:44,050 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:44,050 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:44,050 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,050 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:44,052 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,052 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:44,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:44,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:44,052 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:44,052 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:44,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:44,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:44,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:44,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:44,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:44,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:44,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:44,058 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:44,058 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:44,058 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:44,058 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:44,058 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:44,067 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:44,067 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:44,068 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:44,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:44,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:44,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:44,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:44,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:44,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:44,069 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:44,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:44,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:44,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:44,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:44,070 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:44,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:44,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:44,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,076 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,076 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:44,076 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:44,076 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:44,076 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,076 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:44,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:44,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:44,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:44,076 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:44,078 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:44,078 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:44,076 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:44,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:44,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:44,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:44,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:44,079 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:44,079 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:44,179 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:44,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:44,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:44,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:44,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:44,187 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:44,187 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:44,190 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:44,191 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:44,191 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:44,191 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:44,192 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:44,192 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:44,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:44,192 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,192 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:44,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:44,193 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:44,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:44,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:44,193 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:44,193 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:44,193 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:44,193 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:44,193 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:44,194 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:44,194 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:44,194 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:44,194 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:44,194 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:44,195 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fc3812e05598851193449f5684adf036 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-20 12:53:44,195 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=A 2023-04-20 12:53:44,195 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:53:44,195 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=B 2023-04-20 12:53:44,195 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:53:44,195 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=C 2023-04-20 12:53:44,195 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:53:44,210 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=256 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/536f4132bba14b79b059d6146e90f070 2023-04-20 12:53:44,237 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=256 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/6c6ab749f3ad4ba79733dffe591843d8 2023-04-20 12:53:44,654 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=256 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/8cf2ab1f6be6404fb8aa879543aba93e 2023-04-20 12:53:44,662 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/536f4132bba14b79b059d6146e90f070 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/536f4132bba14b79b059d6146e90f070 2023-04-20 12:53:44,669 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/536f4132bba14b79b059d6146e90f070, entries=150, sequenceid=256, filesize=11.8 K 2023-04-20 12:53:44,672 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/6c6ab749f3ad4ba79733dffe591843d8 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/6c6ab749f3ad4ba79733dffe591843d8 2023-04-20 12:53:44,680 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/6c6ab749f3ad4ba79733dffe591843d8, entries=150, sequenceid=256, filesize=11.8 K 2023-04-20 12:53:44,683 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/8cf2ab1f6be6404fb8aa879543aba93e as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/8cf2ab1f6be6404fb8aa879543aba93e 2023-04-20 12:53:44,694 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/8cf2ab1f6be6404fb8aa879543aba93e, entries=150, sequenceid=256, filesize=11.8 K 2023-04-20 12:53:44,696 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=0 B/0 for fc3812e05598851193449f5684adf036 in 501ms, sequenceid=256, compaction requested=true 2023-04-20 12:53:44,697 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:44,697 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:44,697 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:44,698 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:44,698 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,698 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:44,698 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:44,699 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,700 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:44,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:44,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:44,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:44,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:44,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:44,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,701 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:44,701 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:44,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:44,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:44,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@294edf3f[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:44,702 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:44,702 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:44,703 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:44,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:44,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:44,704 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:44,704 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:44,704 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:44,704 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,704 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:44,705 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:44,705 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:44,705 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:44,705 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:44,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:44,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:44,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:44,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:44,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:44,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:44,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:44,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:44,710 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:44,710 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:44,710 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:44,710 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:44,710 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:44,717 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:44,717 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:44,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:44,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:44,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:44,717 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:44,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:44,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:44,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:44,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:44,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:44,719 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:44,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:44,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:44,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:44,720 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:44,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:44,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,726 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,726 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:44,726 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:44,726 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:44,726 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:44,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:44,726 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:44,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:44,726 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:44,726 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:44,726 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:44,726 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:44,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:44,731 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:44,730 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:44,731 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:44,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:44,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:44,831 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:44,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:44,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:44,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:44,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:44,835 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:44,835 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:44,836 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:44,836 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:44,837 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:44,837 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:44,837 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:44,837 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:44,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:44,838 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,838 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:44,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:44,838 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:44,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:44,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:44,838 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:44,839 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:44,839 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:44,841 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:44,841 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:44,842 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:44,842 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:44,842 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:44,842 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:44,842 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:44,842 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:44,842 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:44,842 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:44,842 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:44,842 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,842 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:44,842 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:44,851 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,851 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:44,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:44,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:44,851 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:44,851 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:44,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:44,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:44,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:44,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:44,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:44,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d47ef2e[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:44,853 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:44,853 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:44,854 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:44,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:44,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:44,855 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:44,855 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:44,855 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,855 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:44,855 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:44,856 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,856 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:44,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:44,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:44,857 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:44,857 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:44,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:44,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:44,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:44,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:44,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:44,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:44,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:44,860 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:44,860 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:44,860 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:44,860 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:44,860 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:44,861 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:44,861 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:44,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:44,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:44,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:44,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:44,861 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:44,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:44,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:44,862 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:44,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:44,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:44,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:44,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:44,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:44,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,863 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:44,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:44,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,865 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,865 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:44,865 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:44,865 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:44,865 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:44,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:44,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:44,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:44,865 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:44,866 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,866 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:44,866 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:44,866 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:44,866 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:44,866 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:44,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:44,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:44,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:44,966 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:44,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:44,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:44,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:44,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:44,970 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:44,970 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:44,971 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:44,971 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:44,972 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:44,972 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:44,973 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,973 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:44,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:44,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:44,973 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:44,973 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:44,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:44,973 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:44,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:44,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:44,974 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:44,974 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:44,974 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:44,974 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:44,974 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:44,975 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:44,976 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:44,976 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:44,976 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:44,976 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:44,976 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fc3812e05598851193449f5684adf036 3/3 column families, dataSize=6.71 KB heapSize=20.67 KB 2023-04-20 12:53:44,976 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=A 2023-04-20 12:53:44,977 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:53:44,980 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=B 2023-04-20 12:53:44,980 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:53:44,981 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=C 2023-04-20 12:53:44,981 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:53:44,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on fc3812e05598851193449f5684adf036 2023-04-20 12:53:44,991 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:45,034 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681995225010, max timestamp=1681995225032Num uniques 859; 2023-04-20 12:53:45,034 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 12:53:45,035 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681995225010, max timestamp=1681995225032Num uniques 859; 2023-04-20 12:53:45,035 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 12:53:45,035 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681995225010, max timestamp=1681995225032Num uniques 859; 2023-04-20 12:53:45,035 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 12:53:45,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681995225049, max timestamp=1681995225056Num uniques 1018; 2023-04-20 12:53:45,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-20 12:53:45,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681995225049, max timestamp=1681995225056Num uniques 1018; 2023-04-20 12:53:45,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-20 12:53:45,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681995225049, max timestamp=1681995225056Num uniques 1018; 2023-04-20 12:53:45,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-20 12:53:45,097 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=377, cellSize=17262, totalHeapSize=39448, min timestamp=1681995225073, max timestamp=1681995225093Num uniques 1127; 2023-04-20 12:53:45,097 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=17262, suffix heap size=145176,new segment heap size=39448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=377 2023-04-20 12:53:45,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681995225082, max timestamp=1681995225093Num uniques 1127; 2023-04-20 12:53:45,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-04-20 12:53:45,099 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681995225082, max timestamp=1681995225093Num uniques 1127; 2023-04-20 12:53:45,099 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-04-20 12:53:45,121 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=486, cellSize=22256, totalHeapSize=50784, min timestamp=1681995225104, max timestamp=1681995225118Num uniques 1236; 2023-04-20 12:53:45,121 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=486, cellSize=22256, totalHeapSize=50784, min timestamp=1681995225104, max timestamp=1681995225118Num uniques 1036; 2023-04-20 12:53:45,121 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=56606, new segment data size=22256, suffix heap size=156512,new segment heap size=50784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1236, new segment cells count=486 2023-04-20 12:53:45,121 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=22256, suffix heap size=135712,new segment heap size=50784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=486 2023-04-20 12:53:45,121 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=486, cellSize=22256, totalHeapSize=50784, min timestamp=1681995225104, max timestamp=1681995225118Num uniques 1036; 2023-04-20 12:53:45,121 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=22256, suffix heap size=135712,new segment heap size=50784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=486 2023-04-20 12:53:45,141 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:45,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32782 deadline: 1681995285137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,141 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:45,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32786 deadline: 1681995285138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,142 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:45,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60994 deadline: 1681995285140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1681995225131, max timestamp=1681995225139Num uniques 1345; 2023-04-20 12:53:45,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1681995225131, max timestamp=1681995225139Num uniques 1345; 2023-04-20 12:53:45,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=61600, new segment data size=15800, suffix heap size=167848,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1345, new segment cells count=345 2023-04-20 12:53:45,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=61600, new segment data size=15800, suffix heap size=167848,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1345, new segment cells count=345 2023-04-20 12:53:45,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1681995225131, max timestamp=1681995225139Num uniques 1345; 2023-04-20 12:53:45,143 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=61600, new segment data size=15800, suffix heap size=167848,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1345, new segment cells count=345 2023-04-20 12:53:45,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681995225179, max timestamp=1681995225200Num uniques 1204; 2023-04-20 12:53:45,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=55140, new segment data size=9340, suffix heap size=153184,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1204, new segment cells count=204 2023-04-20 12:53:45,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681995225179, max timestamp=1681995225200Num uniques 1204; 2023-04-20 12:53:45,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681995225179, max timestamp=1681995225200Num uniques 1204; 2023-04-20 12:53:45,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=55140, new segment data size=9340, suffix heap size=153184,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1204, new segment cells count=204 2023-04-20 12:53:45,203 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=55140, new segment data size=9340, suffix heap size=153184,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1204, new segment cells count=204 2023-04-20 12:53:45,248 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681995225237, max timestamp=1681995225245Num uniques 1063; 2023-04-20 12:53:45,248 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-20 12:53:45,248 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681995225237, max timestamp=1681995225245Num uniques 1063; 2023-04-20 12:53:45,249 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-20 12:53:45,249 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681995225237, max timestamp=1681995225245Num uniques 1063; 2023-04-20 12:53:45,249 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-20 12:53:45,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=422, cellSize=19322, totalHeapSize=44128, min timestamp=1681995225256, max timestamp=1681995225274Num uniques 1072; 2023-04-20 12:53:45,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=422, cellSize=19322, totalHeapSize=44128, min timestamp=1681995225256, max timestamp=1681995225274Num uniques 1072; 2023-04-20 12:53:45,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=19322, suffix heap size=139456,new segment heap size=44128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=422 2023-04-20 12:53:45,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=422, cellSize=19322, totalHeapSize=44128, min timestamp=1681995225256, max timestamp=1681995225274Num uniques 1072; 2023-04-20 12:53:45,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=19322, suffix heap size=139456,new segment heap size=44128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=422 2023-04-20 12:53:45,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=19322, suffix heap size=139456,new segment heap size=44128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=422 2023-04-20 12:53:45,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=331, cellSize=15156, totalHeapSize=34664, min timestamp=1681995225287, max timestamp=1681995225304Num uniques 1281; 2023-04-20 12:53:45,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=58666, new segment data size=15156, suffix heap size=161192,new segment heap size=34664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1281, new segment cells count=331 2023-04-20 12:53:45,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=331, cellSize=15156, totalHeapSize=34664, min timestamp=1681995225287, max timestamp=1681995225304Num uniques 1281; 2023-04-20 12:53:45,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=331, cellSize=15156, totalHeapSize=34664, min timestamp=1681995225287, max timestamp=1681995225304Num uniques 1281; 2023-04-20 12:53:45,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=58666, new segment data size=15156, suffix heap size=161192,new segment heap size=34664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1281, new segment cells count=331 2023-04-20 12:53:45,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=58666, new segment data size=15156, suffix heap size=161192,new segment heap size=34664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1281, new segment cells count=331 2023-04-20 12:53:45,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=390, cellSize=17850, totalHeapSize=40800, min timestamp=1681995225331, max timestamp=1681995225340Num uniques 1190; 2023-04-20 12:53:45,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=390, cellSize=17860, totalHeapSize=40800, min timestamp=1681995225331, max timestamp=1681995225340Num uniques 1190; 2023-04-20 12:53:45,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=54490, new segment data size=17850, suffix heap size=151728,new segment heap size=40800 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1190, new segment cells count=390 2023-04-20 12:53:45,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=54500, new segment data size=17860, suffix heap size=151728,new segment heap size=40800 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1190, new segment cells count=390 2023-04-20 12:53:45,343 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=390, cellSize=17850, totalHeapSize=40800, min timestamp=1681995225331, max timestamp=1681995225340Num uniques 1190; 2023-04-20 12:53:45,343 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=54490, new segment data size=17850, suffix heap size=151728,new segment heap size=40800 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1190, new segment cells count=390 2023-04-20 12:53:45,363 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=349, cellSize=15978, totalHeapSize=36536, min timestamp=1681995225351, max timestamp=1681995225360Num uniques 1249; 2023-04-20 12:53:45,363 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=349, cellSize=15974, totalHeapSize=36536, min timestamp=1681995225351, max timestamp=1681995225360Num uniques 1249; 2023-04-20 12:53:45,363 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=349, cellSize=15974, totalHeapSize=36536, min timestamp=1681995225351, max timestamp=1681995225360Num uniques 1249; 2023-04-20 12:53:45,363 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=57194, new segment data size=15974, suffix heap size=157864,new segment heap size=36536 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1249, new segment cells count=349 2023-04-20 12:53:45,363 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=57198, new segment data size=15978, suffix heap size=157864,new segment heap size=36536 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1249, new segment cells count=349 2023-04-20 12:53:45,363 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=57194, new segment data size=15974, suffix heap size=157864,new segment heap size=36536 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1249, new segment cells count=349 2023-04-20 12:53:45,386 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1681995225379, max timestamp=1681995225383Num uniques 1208; 2023-04-20 12:53:45,386 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1681995225379, max timestamp=1681995225383Num uniques 1208; 2023-04-20 12:53:45,386 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1681995225379, max timestamp=1681995225383Num uniques 1208; 2023-04-20 12:53:45,386 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=55320, new segment data size=9520, suffix heap size=153600,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1208, new segment cells count=208 2023-04-20 12:53:45,386 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=55320, new segment data size=9520, suffix heap size=153600,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1208, new segment cells count=208 2023-04-20 12:53:45,386 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=55320, new segment data size=9520, suffix heap size=153600,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1208, new segment cells count=208 2023-04-20 12:53:45,401 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=261 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/0c1b15029a144d9d98f5a823934c5ba3 2023-04-20 12:53:45,408 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1681995225396, max timestamp=1681995225405Num uniques 1067; 2023-04-20 12:53:45,408 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=9932, suffix heap size=138936,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=217 2023-04-20 12:53:45,410 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1681995225396, max timestamp=1681995225405Num uniques 1067; 2023-04-20 12:53:45,410 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1681995225396, max timestamp=1681995225405Num uniques 1067; 2023-04-20 12:53:45,410 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=7642, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-04-20 12:53:45,410 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=7642, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-04-20 12:53:45,433 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=261 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/45eb80f0dae9487ba746bd611d21bd6a 2023-04-20 12:53:45,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=326, cellSize=14926, totalHeapSize=34144, min timestamp=1681995225411, max timestamp=1681995225441Num uniques 1076; 2023-04-20 12:53:45,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=14926, suffix heap size=139872,new segment heap size=34144 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=326 2023-04-20 12:53:45,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=326, cellSize=14926, totalHeapSize=34144, min timestamp=1681995225411, max timestamp=1681995225441Num uniques 1026; 2023-04-20 12:53:45,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=14926, suffix heap size=134672,new segment heap size=34144 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=326 2023-04-20 12:53:45,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1681995225432, max timestamp=1681995225441Num uniques 1026; 2023-04-20 12:53:45,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=8056, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-04-20 12:53:45,467 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=435, cellSize=19920, totalHeapSize=45480, min timestamp=1681995225451, max timestamp=1681995225464Num uniques 1185; 2023-04-20 12:53:45,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=435, cellSize=19920, totalHeapSize=45480, min timestamp=1681995225451, max timestamp=1681995225464Num uniques 1185; 2023-04-20 12:53:45,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=54270, new segment data size=19920, suffix heap size=151208,new segment heap size=45480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1185, new segment cells count=435 2023-04-20 12:53:45,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=54270, new segment data size=19920, suffix heap size=151208,new segment heap size=45480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1185, new segment cells count=435 2023-04-20 12:53:45,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=435, cellSize=19920, totalHeapSize=45480, min timestamp=1681995225451, max timestamp=1681995225464Num uniques 1035; 2023-04-20 12:53:45,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=19920, suffix heap size=135608,new segment heap size=45480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=435 2023-04-20 12:53:45,482 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=261 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/019aa894e9a541b4b205239822ff57de 2023-04-20 12:53:45,491 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/0c1b15029a144d9d98f5a823934c5ba3 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/0c1b15029a144d9d98f5a823934c5ba3 2023-04-20 12:53:45,498 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/0c1b15029a144d9d98f5a823934c5ba3, entries=50, sequenceid=261, filesize=7.1 K 2023-04-20 12:53:45,501 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/45eb80f0dae9487ba746bd611d21bd6a as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/45eb80f0dae9487ba746bd611d21bd6a 2023-04-20 12:53:45,507 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8874, totalHeapSize=20416, min timestamp=1681995225488, max timestamp=1681995225495Num uniques 1294; 2023-04-20 12:53:45,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=59254, new segment data size=8874, suffix heap size=162544,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1294, new segment cells count=194 2023-04-20 12:53:45,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8864, totalHeapSize=20416, min timestamp=1681995225488, max timestamp=1681995225499Num uniques 1294; 2023-04-20 12:53:45,509 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8874, totalHeapSize=20416, min timestamp=1681995225488, max timestamp=1681995225495Num uniques 1294; 2023-04-20 12:53:45,508 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:45,509 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=59254, new segment data size=8874, suffix heap size=162544,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1294, new segment cells count=194 2023-04-20 12:53:45,509 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=59244, new segment data size=8864, suffix heap size=162544,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1294, new segment cells count=194 2023-04-20 12:53:45,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32832 deadline: 1681995285492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,516 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/45eb80f0dae9487ba746bd611d21bd6a, entries=50, sequenceid=261, filesize=7.1 K 2023-04-20 12:53:45,519 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/019aa894e9a541b4b205239822ff57de as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/019aa894e9a541b4b205239822ff57de 2023-04-20 12:53:45,527 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/019aa894e9a541b4b205239822ff57de, entries=50, sequenceid=261, filesize=7.1 K 2023-04-20 12:53:45,529 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~6.71 KB/6870, heapSize ~20.63 KB/21120, currentSize=73.80 KB/75570 for fc3812e05598851193449f5684adf036 in 552ms, sequenceid=261, compaction requested=true 2023-04-20 12:53:45,529 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:45,529 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:45,530 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:45,530 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:45,530 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,531 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:45,531 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:45,532 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,532 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:45,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:45,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:45,532 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:45,532 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:45,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:45,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:45,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:45,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:45,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:45,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30b0eca0[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:45,534 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:45,534 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:45,535 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:45,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:45,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:45,535 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:45,535 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:45,535 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:45,535 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,535 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:45,536 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:45,536 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,536 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:45,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:45,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:45,536 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:45,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:45,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on fc3812e05598851193449f5684adf036 2023-04-20 12:53:45,539 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fc3812e05598851193449f5684adf036 3/3 column families, dataSize=80.51 KB heapSize=222.33 KB 2023-04-20 12:53:45,539 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=A 2023-04-20 12:53:45,539 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:53:45,539 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=B 2023-04-20 12:53:45,539 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:53:45,539 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=C 2023-04-20 12:53:45,539 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:53:45,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:45,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:45,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:45,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:45,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:45,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:45,543 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:45,543 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:45,543 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:45,543 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:45,544 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:45,551 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:45,551 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:45,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:45,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:45,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:45,551 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:45,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:45,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:45,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:45,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:45,552 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:45,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:45,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:45,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:45,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:45,553 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:45,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:45,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,559 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,559 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:45,559 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:45,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:45,559 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:45,559 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:45,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:45,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:45,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:45,559 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:45,561 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:45,561 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:45,562 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:45,561 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,562 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:45,562 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:45,562 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:45,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:45,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:45,569 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:45,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32792 deadline: 1681995285565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,570 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:45,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32786 deadline: 1681995285565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,572 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:45,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32782 deadline: 1681995285567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,572 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:45,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60994 deadline: 1681995285568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,586 WARN [DataStreamer for file /user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/103039a0ca884bf39bd298e171bb47b9] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-04-20 12:53:45,586 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=547 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/103039a0ca884bf39bd298e171bb47b9 2023-04-20 12:53:45,612 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:45,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32832 deadline: 1681995285611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,618 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=547 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/f6e05bfe8ed74791804c16123ac6ad79 2023-04-20 12:53:45,633 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=547 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/78ccd2a3ffb043f0b7765caeb6cb1b5a 2023-04-20 12:53:45,640 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/103039a0ca884bf39bd298e171bb47b9 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/103039a0ca884bf39bd298e171bb47b9 2023-04-20 12:53:45,646 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/103039a0ca884bf39bd298e171bb47b9, entries=450, sequenceid=547, filesize=26.2 K 2023-04-20 12:53:45,648 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/f6e05bfe8ed74791804c16123ac6ad79 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/f6e05bfe8ed74791804c16123ac6ad79 2023-04-20 12:53:45,655 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/f6e05bfe8ed74791804c16123ac6ad79, entries=150, sequenceid=547, filesize=11.9 K 2023-04-20 12:53:45,659 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/78ccd2a3ffb043f0b7765caeb6cb1b5a as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/78ccd2a3ffb043f0b7765caeb6cb1b5a 2023-04-20 12:53:45,662 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:45,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:45,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:45,666 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/78ccd2a3ffb043f0b7765caeb6cb1b5a, entries=150, sequenceid=547, filesize=11.9 K 2023-04-20 12:53:45,667 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~222.28 KB/227616, currentSize=100.63 KB/103050 for fc3812e05598851193449f5684adf036 in 128ms, sequenceid=547, compaction requested=true 2023-04-20 12:53:45,667 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:45,667 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:53:45,667 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-20 12:53:45,667 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-20 12:53:45,667 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:53:45,667 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:53:45,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:45,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:45,667 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:45,667 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:45,669 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 82480 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-20 12:53:45,669 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 67836 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-20 12:53:45,670 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): fc3812e05598851193449f5684adf036/A is initiating minor compaction (all files) 2023-04-20 12:53:45,670 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): fc3812e05598851193449f5684adf036/B is initiating minor compaction (all files) 2023-04-20 12:53:45,670 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fc3812e05598851193449f5684adf036/A in TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:45,670 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fc3812e05598851193449f5684adf036/B in TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:45,670 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/15bd28d10e954c4d96a60c94428f59d5, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/b169c2caecc4494bafed10149a40d74d, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/d2510056be1846ef912b783cbf62405e, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/536f4132bba14b79b059d6146e90f070, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/0c1b15029a144d9d98f5a823934c5ba3, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/103039a0ca884bf39bd298e171bb47b9] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp, totalSize=80.5 K 2023-04-20 12:53:45,670 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/d303006629324fbcad8537dd27e10df4, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/b2527240d2ca4f2f8d735a88d4aa7835, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/51f1b23a2d8f4dde9c1879ea1c8bf37f, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/6c6ab749f3ad4ba79733dffe591843d8, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/45eb80f0dae9487ba746bd611d21bd6a, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/f6e05bfe8ed74791804c16123ac6ad79] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp, totalSize=66.2 K 2023-04-20 12:53:45,671 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting d303006629324fbcad8537dd27e10df4, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1681995220780 2023-04-20 12:53:45,671 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 15bd28d10e954c4d96a60c94428f59d5, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1681995220780 2023-04-20 12:53:45,673 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:45,673 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:45,672 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting b2527240d2ca4f2f8d735a88d4aa7835, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=207, earliestPutTs=1681995221028 2023-04-20 12:53:45,674 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:45,674 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:45,674 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting b169c2caecc4494bafed10149a40d74d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=207, earliestPutTs=1681995221028 2023-04-20 12:53:45,674 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 51f1b23a2d8f4dde9c1879ea1c8bf37f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=221, earliestPutTs=1681995221688 2023-04-20 12:53:45,675 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 6c6ab749f3ad4ba79733dffe591843d8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=256, earliestPutTs=1681995222861 2023-04-20 12:53:45,675 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,675 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:45,675 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting d2510056be1846ef912b783cbf62405e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=221, earliestPutTs=1681995221688 2023-04-20 12:53:45,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:45,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:45,676 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,676 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:45,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:45,676 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 536f4132bba14b79b059d6146e90f070, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=256, earliestPutTs=1681995222861 2023-04-20 12:53:45,676 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 45eb80f0dae9487ba746bd611d21bd6a, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1681995224974 2023-04-20 12:53:45,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:45,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on fc3812e05598851193449f5684adf036 2023-04-20 12:53:45,677 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 0c1b15029a144d9d98f5a823934c5ba3, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1681995224974 2023-04-20 12:53:45,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:45,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:45,677 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting f6e05bfe8ed74791804c16123ac6ad79, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=547, earliestPutTs=1681995225495 2023-04-20 12:53:45,677 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fc3812e05598851193449f5684adf036 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-20 12:53:45,677 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:45,677 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:45,677 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:45,677 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:45,677 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:45,680 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:45,680 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 103039a0ca884bf39bd298e171bb47b9, keycount=450, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=547, earliestPutTs=1681995225490 2023-04-20 12:53:45,680 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:45,680 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:45,680 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:45,680 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:45,680 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:45,680 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:45,680 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:45,680 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:45,680 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=A 2023-04-20 12:53:45,680 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,681 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:45,681 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:53:45,681 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:45,681 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=B 2023-04-20 12:53:45,681 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:53:45,682 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=C 2023-04-20 12:53:45,682 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:53:45,688 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,688 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:45,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:45,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:45,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:45,689 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:45,690 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:45,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:45,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:45,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:45,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:45,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c52ba4a[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:45,692 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:45,692 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:45,696 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:45,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:45,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:45,696 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:45,696 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:45,697 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:45,697 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,697 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:45,698 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:45,698 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:45,698 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:45,698 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:45,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:45,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:45,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:45,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:45,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:45,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:45,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:45,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:45,702 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:45,702 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:45,702 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:45,702 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:45,702 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:45,704 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): fc3812e05598851193449f5684adf036#B#compaction#36 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:53:45,704 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:45,705 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:45,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:45,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:45,705 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:45,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:45,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:45,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:45,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:45,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:45,706 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:45,706 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:45,706 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:45,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:45,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32792 deadline: 1681995285695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60994 deadline: 1681995285693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:45,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:45,707 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:45,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:45,710 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,710 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:45,710 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:45,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:45,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:45,710 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,710 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:45,710 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:45,710 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:45,710 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:45,710 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:45,710 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:45,713 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:45,713 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:45,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:45,713 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): fc3812e05598851193449f5684adf036#A#compaction#37 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:53:45,710 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:45,713 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:45,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:45,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:45,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:45,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:45,714 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:45,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32782 deadline: 1681995285707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,715 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:45,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32786 deadline: 1681995285707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,717 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=567 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/bb63fee100424f148013afe63ee26fcd 2023-04-20 12:53:45,742 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/ce4a2b24f0ff4e568b435ac022ab24e2 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/ce4a2b24f0ff4e568b435ac022ab24e2 2023-04-20 12:53:45,750 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in fc3812e05598851193449f5684adf036/B of fc3812e05598851193449f5684adf036 into ce4a2b24f0ff4e568b435ac022ab24e2(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:53:45,751 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:45,751 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036., storeName=fc3812e05598851193449f5684adf036/B, priority=10, startTime=1681995225667; duration=0sec 2023-04-20 12:53:45,751 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:53:45,752 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-20 12:53:45,755 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=567 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/a15ffd1b73414cce82dd63db6a945b67 2023-04-20 12:53:45,759 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 79804 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-20 12:53:45,759 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): fc3812e05598851193449f5684adf036/C is initiating minor compaction (all files) 2023-04-20 12:53:45,760 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fc3812e05598851193449f5684adf036/C in TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:45,760 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/0b26314a595f497cad9b2f78ecc05890, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/4d56472ab5dc43c7a7945e2605ec59ca, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/523c7f28b3bd476e8d3bf0dcfa8ba35f, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/254fbb5994d74ac1bd4886aa60a4f25c, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/8cf2ab1f6be6404fb8aa879543aba93e, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/019aa894e9a541b4b205239822ff57de, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/78ccd2a3ffb043f0b7765caeb6cb1b5a] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp, totalSize=77.9 K 2023-04-20 12:53:45,760 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 0b26314a595f497cad9b2f78ecc05890, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1681995219641 2023-04-20 12:53:45,761 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 4d56472ab5dc43c7a7945e2605ec59ca, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1681995220780 2023-04-20 12:53:45,761 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 523c7f28b3bd476e8d3bf0dcfa8ba35f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=207, earliestPutTs=1681995221028 2023-04-20 12:53:45,762 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 254fbb5994d74ac1bd4886aa60a4f25c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=221, earliestPutTs=1681995221688 2023-04-20 12:53:45,763 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 8cf2ab1f6be6404fb8aa879543aba93e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=256, earliestPutTs=1681995222861 2023-04-20 12:53:45,764 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 019aa894e9a541b4b205239822ff57de, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1681995224974 2023-04-20 12:53:45,765 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 78ccd2a3ffb043f0b7765caeb6cb1b5a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=547, earliestPutTs=1681995225495 2023-04-20 12:53:45,784 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): fc3812e05598851193449f5684adf036#C#compaction#40 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:53:45,793 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=567 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/88e09a6207ea48948d108830316f4167 2023-04-20 12:53:45,799 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/bb63fee100424f148013afe63ee26fcd as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/bb63fee100424f148013afe63ee26fcd 2023-04-20 12:53:45,809 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:45,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60994 deadline: 1681995285808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:45,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32792 deadline: 1681995285811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,813 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:45,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:45,817 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/bb63fee100424f148013afe63ee26fcd, entries=200, sequenceid=567, filesize=14.3 K 2023-04-20 12:53:45,817 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:45,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32832 deadline: 1681995285813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:45,818 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/a15ffd1b73414cce82dd63db6a945b67 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/a15ffd1b73414cce82dd63db6a945b67 2023-04-20 12:53:45,820 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:45,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32786 deadline: 1681995285817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,821 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:45,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32782 deadline: 1681995285817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,825 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/a15ffd1b73414cce82dd63db6a945b67, entries=150, sequenceid=567, filesize=11.9 K 2023-04-20 12:53:45,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:45,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:45,828 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:45,828 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:45,828 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/88e09a6207ea48948d108830316f4167 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/88e09a6207ea48948d108830316f4167 2023-04-20 12:53:45,828 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:45,828 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:45,829 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:45,829 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:45,829 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,830 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,830 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:45,830 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:45,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:45,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:45,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:45,831 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:45,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:45,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:45,831 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:45,832 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:45,832 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:45,832 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:45,832 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:45,832 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:45,832 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:45,832 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:45,832 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:45,832 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:45,832 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:45,832 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:45,832 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:45,833 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:45,833 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,833 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:45,833 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:45,835 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,835 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:45,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:45,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:45,836 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:45,836 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:45,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:45,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:45,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:45,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:45,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:45,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23ba35d8[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:45,837 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:45,838 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:45,843 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,843 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:45,843 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/88e09a6207ea48948d108830316f4167, entries=150, sequenceid=567, filesize=11.9 K 2023-04-20 12:53:45,844 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:45,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:45,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:45,844 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:45,844 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:45,844 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:45,845 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for fc3812e05598851193449f5684adf036 in 168ms, sequenceid=567, compaction requested=false 2023-04-20 12:53:45,845 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:45,854 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:45,854 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,855 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:45,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:45,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:45,855 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:45,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:45,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:45,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:45,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:45,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:45,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:45,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:45,858 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:45,858 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:45,858 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:45,858 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:45,858 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:45,859 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:45,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:45,859 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:45,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:45,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:45,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:45,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:45,859 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:45,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:45,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:45,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:45,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:45,861 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:45,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:45,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:45,862 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:45,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:45,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,868 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,868 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:45,868 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:45,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:45,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:45,868 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:45,868 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:45,869 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,869 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:45,869 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:45,869 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:45,869 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:45,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:45,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:45,870 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:45,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:45,870 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:45,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:45,970 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:45,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:45,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:45,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:45,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:45,974 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:45,974 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:45,974 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:45,974 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:45,975 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:45,975 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:45,978 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,978 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:45,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:45,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:45,979 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:45,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:45,979 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:45,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:45,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:45,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:45,979 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:45,980 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:45,980 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:45,980 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:45,980 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:45,980 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:45,980 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:45,980 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:45,980 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:45,980 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:45,981 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fc3812e05598851193449f5684adf036 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-20 12:53:45,981 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=A 2023-04-20 12:53:45,981 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:53:45,981 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=B 2023-04-20 12:53:45,981 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:53:45,981 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=C 2023-04-20 12:53:45,981 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:53:45,993 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=580 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/4a426bcc9c964cc49ed1465eaa014840 2023-04-20 12:53:46,014 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:46,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on fc3812e05598851193449f5684adf036 2023-04-20 12:53:46,029 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=580 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/7264c8e180ac4b20b5d64aecc562ad61 2023-04-20 12:53:46,090 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:46,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32786 deadline: 1681995286059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,090 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:46,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32792 deadline: 1681995286085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,091 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:46,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32782 deadline: 1681995286085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,091 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:46,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60994 deadline: 1681995286085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,121 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:46,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:32832 deadline: 1681995286120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,153 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/8932caa62d6544a39c1bc2521de7ebc7 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/8932caa62d6544a39c1bc2521de7ebc7 2023-04-20 12:53:46,162 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in fc3812e05598851193449f5684adf036/A of fc3812e05598851193449f5684adf036 into 8932caa62d6544a39c1bc2521de7ebc7(size=12.3 K), total size for store is 26.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:53:46,162 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:46,162 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036., storeName=fc3812e05598851193449f5684adf036/A, priority=10, startTime=1681995225667; duration=0sec 2023-04-20 12:53:46,162 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:53:46,192 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:46,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32792 deadline: 1681995286192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,195 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:46,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32786 deadline: 1681995286194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,196 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:46,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60994 deadline: 1681995286194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,197 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:46,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32782 deadline: 1681995286195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,202 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/c8376418c06840e28cd81a1b5668fe60 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/c8376418c06840e28cd81a1b5668fe60 2023-04-20 12:53:46,213 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in fc3812e05598851193449f5684adf036/C of fc3812e05598851193449f5684adf036 into c8376418c06840e28cd81a1b5668fe60(size=12.3 K), total size for store is 24.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:53:46,213 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:46,213 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036., storeName=fc3812e05598851193449f5684adf036/C, priority=9, startTime=1681995225667; duration=0sec 2023-04-20 12:53:46,213 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:53:46,396 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:46,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32792 deadline: 1681995286394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,399 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:46,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32786 deadline: 1681995286397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,400 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:46,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60994 deadline: 1681995286398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,400 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:46,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32782 deadline: 1681995286399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,455 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=580 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/f5b7df47a1ac42938efc566203740c93 2023-04-20 12:53:46,461 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/4a426bcc9c964cc49ed1465eaa014840 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/4a426bcc9c964cc49ed1465eaa014840 2023-04-20 12:53:46,470 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/4a426bcc9c964cc49ed1465eaa014840, entries=150, sequenceid=580, filesize=11.9 K 2023-04-20 12:53:46,471 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/7264c8e180ac4b20b5d64aecc562ad61 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/7264c8e180ac4b20b5d64aecc562ad61 2023-04-20 12:53:46,478 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/7264c8e180ac4b20b5d64aecc562ad61, entries=150, sequenceid=580, filesize=11.9 K 2023-04-20 12:53:46,479 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/f5b7df47a1ac42938efc566203740c93 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/f5b7df47a1ac42938efc566203740c93 2023-04-20 12:53:46,487 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/f5b7df47a1ac42938efc566203740c93, entries=150, sequenceid=580, filesize=11.9 K 2023-04-20 12:53:46,488 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for fc3812e05598851193449f5684adf036 in 508ms, sequenceid=580, compaction requested=true 2023-04-20 12:53:46,488 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:46,488 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:46,488 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:46,488 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:46,488 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,489 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:46,489 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:46,493 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:46,493 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:46,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:46,493 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:46,493 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:46,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:46,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:46,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:46,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:46,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:46,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4166801e[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:46,494 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:46,494 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:46,495 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:46,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:46,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:46,495 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:46,495 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:46,495 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:46,495 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,495 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:46,497 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:46,497 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,497 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:46,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:46,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:46,513 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:46,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:46,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:46,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:46,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:46,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:46,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:46,522 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:46,522 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:46,523 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:46,523 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:46,523 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:46,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:46,524 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:46,524 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:46,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:46,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:46,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:46,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:46,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:46,524 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:46,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:46,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:46,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:46,525 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:46,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:46,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:46,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:46,526 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:46,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:46,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,534 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,534 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:46,534 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:46,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:46,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:46,534 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,535 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:46,535 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:46,535 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:46,534 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:46,535 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:46,534 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:46,535 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:46,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:46,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:46,535 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:46,535 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:46,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:46,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:46,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on fc3812e05598851193449f5684adf036 2023-04-20 12:53:46,625 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fc3812e05598851193449f5684adf036 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-20 12:53:46,625 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=A 2023-04-20 12:53:46,625 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:53:46,625 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=B 2023-04-20 12:53:46,625 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:53:46,625 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=C 2023-04-20 12:53:46,625 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:53:46,635 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:46,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:46,637 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=603 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/0297843f356f479cba7ba7e6449b8894 2023-04-20 12:53:46,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:46,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:46,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:46,641 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:46,641 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:46,641 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:46,641 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:46,643 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:46,643 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:46,645 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:46,645 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:46,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:46,645 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:46,645 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:46,646 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:46,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:46,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:46,646 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:46,647 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:46,647 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:46,647 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:46,647 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:46,647 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:46,650 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:46,650 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:46,650 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:46,650 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:46,650 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:46,650 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:46,650 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:46,650 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:46,650 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,650 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:46,650 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:46,659 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:46,660 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:46,660 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:46,660 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:46,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:46,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:46,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:46,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:46,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:46,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:46,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7684b17e[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:46,662 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:46,662 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:46,663 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:46,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:46,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:46,663 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:46,663 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:46,663 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:46,663 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,663 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:46,664 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,664 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:46,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:46,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:46,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32832 deadline: 1681995286660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,665 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:46,665 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:46,665 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:46,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:46,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:46,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:46,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:46,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:46,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:46,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:46,667 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:46,667 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:46,667 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:46,667 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:46,667 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:46,668 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:46,668 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:46,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:46,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:46,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:46,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:46,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:46,668 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:46,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:46,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:46,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:46,669 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:46,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:46,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:46,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:46,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,670 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:46,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:46,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,672 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,672 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:46,672 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:46,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:46,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:46,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:46,672 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:46,672 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:46,672 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:46,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:46,672 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:46,672 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:46,673 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:46,672 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:46,673 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:46,673 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:46,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:46,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:46,681 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=603 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/f38625ab486645f3bfa7076eb9145abe 2023-04-20 12:53:46,700 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:46,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32792 deadline: 1681995286699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,704 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:46,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32782 deadline: 1681995286702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,704 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:46,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32786 deadline: 1681995286703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,704 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:46,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60994 deadline: 1681995286703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,767 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:46,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32832 deadline: 1681995286766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,773 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:46,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:46,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:46,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:46,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:46,776 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:46,776 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:46,776 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:46,776 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:46,777 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:46,777 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:46,777 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,777 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:46,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:46,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:46,778 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,778 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:46,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:46,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:46,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:46,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:46,779 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:46,789 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:46,789 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:46,789 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:46,789 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:46,789 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:46,789 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:46,789 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:46,789 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:46,789 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:46,790 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:46,790 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:46,790 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:46,790 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:46,790 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,790 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:46,790 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:46,792 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:46,792 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:46,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:46,793 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:46,793 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:46,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:46,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:46,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:46,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:46,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:46,794 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:46,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f6dd50e[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:46,794 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:46,795 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:46,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:46,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:46,795 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:46,795 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,795 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:46,795 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:46,795 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:46,796 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,796 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:46,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:46,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:46,796 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:46,796 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:46,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:46,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:46,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:46,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:46,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:46,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:46,799 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:46,799 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:46,799 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:46,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:46,799 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:46,799 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:46,800 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:46,800 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:46,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:46,800 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:46,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:46,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:46,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:46,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:46,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:46,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:46,801 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:46,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:46,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:46,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:46,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:46,803 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:46,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:46,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,805 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,805 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:46,805 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:46,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:46,805 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:46,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:46,805 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:46,806 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,806 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:46,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:46,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:46,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:46,805 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:46,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:46,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:46,806 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:46,806 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:46,806 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:46,806 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:46,906 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:46,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:46,909 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:46,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:46,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:46,910 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:46,910 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:46,910 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:46,910 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:46,910 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:46,910 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:46,911 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,911 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:46,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:46,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:46,911 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,911 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:46,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:46,912 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:46,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:46,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:46,912 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:46,913 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:46,913 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:46,913 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:46,913 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:46,913 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:46,913 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:46,913 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:46,913 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:46,913 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:46,913 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:46,913 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:46,913 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:46,913 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:46,913 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,913 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:46,913 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:46,914 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,914 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:46,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:46,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:46,915 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:46,915 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:46,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:46,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:46,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:46,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:46,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:46,916 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:46,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d0b37ce[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:46,916 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:46,916 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:46,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:46,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:46,916 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:46,917 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:46,917 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,917 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:46,917 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:46,917 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,917 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:46,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:46,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:46,917 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:46,918 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:46,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:46,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:46,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:46,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:46,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:46,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:46,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:46,922 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:46,922 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:46,922 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:46,922 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:46,922 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:46,923 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:46,923 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:46,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:46,923 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:46,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:46,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:46,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:46,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:46,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:46,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:46,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:46,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:46,924 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:46,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:46,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:46,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,925 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:46,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:46,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,927 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,927 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:46,927 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:46,927 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:46,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:46,927 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:46,927 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:46,927 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:46,927 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:46,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:46,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:46,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:46,927 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:46,927 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:46,928 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:46,928 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:46,928 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:46,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:46,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:46,969 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:46,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32832 deadline: 1681995286968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,028 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:47,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:47,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:47,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:47,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:47,032 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:47,032 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:47,032 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:47,032 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:47,032 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:47,032 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:47,033 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:47,033 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:47,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:47,033 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,033 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:47,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:47,033 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:47,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:47,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:47,034 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:47,034 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:47,034 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:47,034 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:47,034 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:47,034 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:47,034 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:47,034 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:47,034 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:47,035 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:47,035 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:47,035 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:47,035 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:47,035 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:47,035 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,035 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:47,035 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:47,036 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,036 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:47,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:47,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:47,036 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:47,036 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:47,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:47,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:47,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:47,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:47,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:47,037 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:47,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18182c01[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:47,037 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:47,038 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:47,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:47,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:47,038 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:47,038 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:47,038 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,038 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:47,038 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:47,039 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,039 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:47,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:47,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:47,039 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:47,040 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:47,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:47,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:47,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:47,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:47,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:47,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:47,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:47,041 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:47,042 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:47,042 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:47,042 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:47,042 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:47,043 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:47,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:47,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:47,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:47,043 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:47,043 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:47,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:47,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:47,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:47,043 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:47,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:47,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:47,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:47,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:47,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:47,044 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:47,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:47,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,046 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,046 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:47,046 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:47,046 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:47,046 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,046 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:47,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:47,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:47,047 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:47,047 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:47,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:47,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:47,047 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:47,047 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:47,047 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:47,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:47,047 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:47,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:47,047 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:47,101 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=603 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/3e1faa50ac584b8abda498bb572312b2 2023-04-20 12:53:47,108 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/0297843f356f479cba7ba7e6449b8894 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/0297843f356f479cba7ba7e6449b8894 2023-04-20 12:53:47,113 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/0297843f356f479cba7ba7e6449b8894, entries=200, sequenceid=603, filesize=14.3 K 2023-04-20 12:53:47,115 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/f38625ab486645f3bfa7076eb9145abe as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/f38625ab486645f3bfa7076eb9145abe 2023-04-20 12:53:47,121 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/f38625ab486645f3bfa7076eb9145abe, entries=150, sequenceid=603, filesize=11.9 K 2023-04-20 12:53:47,123 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/3e1faa50ac584b8abda498bb572312b2 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/3e1faa50ac584b8abda498bb572312b2 2023-04-20 12:53:47,129 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/3e1faa50ac584b8abda498bb572312b2, entries=150, sequenceid=603, filesize=11.9 K 2023-04-20 12:53:47,130 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for fc3812e05598851193449f5684adf036 in 505ms, sequenceid=603, compaction requested=true 2023-04-20 12:53:47,130 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:47,130 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:53:47,130 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 12:53:47,131 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 12:53:47,130 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:53:47,131 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:53:47,132 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49154 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 12:53:47,132 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54034 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 12:53:47,133 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): fc3812e05598851193449f5684adf036/B is initiating minor compaction (all files) 2023-04-20 12:53:47,133 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): fc3812e05598851193449f5684adf036/A is initiating minor compaction (all files) 2023-04-20 12:53:47,133 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fc3812e05598851193449f5684adf036/B in TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:47,133 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fc3812e05598851193449f5684adf036/A in TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:47,133 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/ce4a2b24f0ff4e568b435ac022ab24e2, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/a15ffd1b73414cce82dd63db6a945b67, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/7264c8e180ac4b20b5d64aecc562ad61, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/f38625ab486645f3bfa7076eb9145abe] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp, totalSize=48.0 K 2023-04-20 12:53:47,133 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/8932caa62d6544a39c1bc2521de7ebc7, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/bb63fee100424f148013afe63ee26fcd, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/4a426bcc9c964cc49ed1465eaa014840, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/0297843f356f479cba7ba7e6449b8894] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp, totalSize=52.8 K 2023-04-20 12:53:47,133 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting ce4a2b24f0ff4e568b435ac022ab24e2, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=547, earliestPutTs=1681995225495 2023-04-20 12:53:47,134 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 8932caa62d6544a39c1bc2521de7ebc7, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=547, earliestPutTs=1681995225495 2023-04-20 12:53:47,135 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting a15ffd1b73414cce82dd63db6a945b67, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=567, earliestPutTs=1681995225564 2023-04-20 12:53:47,135 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting bb63fee100424f148013afe63ee26fcd, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=567, earliestPutTs=1681995225564 2023-04-20 12:53:47,135 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 7264c8e180ac4b20b5d64aecc562ad61, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=580, earliestPutTs=1681995225685 2023-04-20 12:53:47,136 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 4a426bcc9c964cc49ed1465eaa014840, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=580, earliestPutTs=1681995225685 2023-04-20 12:53:47,136 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting f38625ab486645f3bfa7076eb9145abe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=603, earliestPutTs=1681995226044 2023-04-20 12:53:47,136 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 0297843f356f479cba7ba7e6449b8894, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=603, earliestPutTs=1681995226039 2023-04-20 12:53:47,147 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): fc3812e05598851193449f5684adf036#A#compaction#47 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:53:47,147 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:47,148 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): fc3812e05598851193449f5684adf036#B#compaction#48 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:53:47,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:47,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:47,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:47,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:47,155 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:47,155 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:47,155 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:47,155 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:47,155 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:47,155 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:47,156 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:47,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:47,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:47,156 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:47,157 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,157 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:47,157 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:47,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:47,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:47,161 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:47,162 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:47,162 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:47,162 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:47,162 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:47,162 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:47,162 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:47,162 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:47,162 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/d80b97f269514bbb8233513d9e628405 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/d80b97f269514bbb8233513d9e628405 2023-04-20 12:53:47,162 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:47,163 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:47,163 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fc3812e05598851193449f5684adf036 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-20 12:53:47,163 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=A 2023-04-20 12:53:47,163 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:53:47,163 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=B 2023-04-20 12:53:47,163 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:53:47,164 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=C 2023-04-20 12:53:47,164 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:53:47,169 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in fc3812e05598851193449f5684adf036/A of fc3812e05598851193449f5684adf036 into d80b97f269514bbb8233513d9e628405(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:53:47,170 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:47,170 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036., storeName=fc3812e05598851193449f5684adf036/A, priority=12, startTime=1681995227130; duration=0sec 2023-04-20 12:53:47,170 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:53:47,170 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 12:53:47,172 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49120 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 12:53:47,172 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): fc3812e05598851193449f5684adf036/C is initiating minor compaction (all files) 2023-04-20 12:53:47,172 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fc3812e05598851193449f5684adf036/C in TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:47,172 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/c8376418c06840e28cd81a1b5668fe60, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/88e09a6207ea48948d108830316f4167, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/f5b7df47a1ac42938efc566203740c93, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/3e1faa50ac584b8abda498bb572312b2] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp, totalSize=48.0 K 2023-04-20 12:53:47,173 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting c8376418c06840e28cd81a1b5668fe60, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=547, earliestPutTs=1681995225495 2023-04-20 12:53:47,173 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=614 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/35b85ba6987449c284859e423ea2557b 2023-04-20 12:53:47,173 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 88e09a6207ea48948d108830316f4167, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=567, earliestPutTs=1681995225564 2023-04-20 12:53:47,174 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting f5b7df47a1ac42938efc566203740c93, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=580, earliestPutTs=1681995225685 2023-04-20 12:53:47,174 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 3e1faa50ac584b8abda498bb572312b2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=603, earliestPutTs=1681995226044 2023-04-20 12:53:47,188 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): fc3812e05598851193449f5684adf036#C#compaction#51 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:53:47,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on fc3812e05598851193449f5684adf036 2023-04-20 12:53:47,204 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:47,225 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:47,226 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681995227217, max timestamp=1681995227224Num uniques 859; 2023-04-20 12:53:47,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60994 deadline: 1681995287221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,226 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681995227217, max timestamp=1681995227224Num uniques 859; 2023-04-20 12:53:47,226 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 12:53:47,226 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 12:53:47,226 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681995227217, max timestamp=1681995227224Num uniques 859; 2023-04-20 12:53:47,227 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 12:53:47,256 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:47,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32786 deadline: 1681995287253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,256 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:47,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32792 deadline: 1681995287254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,259 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:47,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32782 deadline: 1681995287255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,272 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:47,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32832 deadline: 1681995287272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,330 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:47,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60994 deadline: 1681995287330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,358 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:47,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32786 deadline: 1681995287358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,359 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:47,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32792 deadline: 1681995287358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,368 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:47,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32782 deadline: 1681995287363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,534 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:47,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60994 deadline: 1681995287532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,562 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:47,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32786 deadline: 1681995287560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,563 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:47,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32792 deadline: 1681995287561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,563 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/dfcb85b67759440f8d1526b85663e3cc as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/dfcb85b67759440f8d1526b85663e3cc 2023-04-20 12:53:47,568 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in fc3812e05598851193449f5684adf036/B of fc3812e05598851193449f5684adf036 into dfcb85b67759440f8d1526b85663e3cc(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:53:47,568 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:47,568 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036., storeName=fc3812e05598851193449f5684adf036/B, priority=12, startTime=1681995227130; duration=0sec 2023-04-20 12:53:47,569 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:53:47,573 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:47,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32782 deadline: 1681995287570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,598 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=614 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/b9bccce7e7b14ca98390dd4f58931c86 2023-04-20 12:53:47,606 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/c638e4cbfb4141f281f070ea1885ae0f as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/c638e4cbfb4141f281f070ea1885ae0f 2023-04-20 12:53:47,613 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in fc3812e05598851193449f5684adf036/C of fc3812e05598851193449f5684adf036 into c638e4cbfb4141f281f070ea1885ae0f(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:53:47,613 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:47,613 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036., storeName=fc3812e05598851193449f5684adf036/C, priority=12, startTime=1681995227131; duration=0sec 2023-04-20 12:53:47,613 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:53:47,626 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=614 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/10c83a7919ff438fb1e051e13ff01a1d 2023-04-20 12:53:47,632 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/35b85ba6987449c284859e423ea2557b as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/35b85ba6987449c284859e423ea2557b 2023-04-20 12:53:47,640 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/35b85ba6987449c284859e423ea2557b, entries=150, sequenceid=614, filesize=11.9 K 2023-04-20 12:53:47,642 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/b9bccce7e7b14ca98390dd4f58931c86 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/b9bccce7e7b14ca98390dd4f58931c86 2023-04-20 12:53:47,648 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/b9bccce7e7b14ca98390dd4f58931c86, entries=150, sequenceid=614, filesize=11.9 K 2023-04-20 12:53:47,655 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/10c83a7919ff438fb1e051e13ff01a1d as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/10c83a7919ff438fb1e051e13ff01a1d 2023-04-20 12:53:47,669 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/10c83a7919ff438fb1e051e13ff01a1d, entries=150, sequenceid=614, filesize=11.9 K 2023-04-20 12:53:47,670 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for fc3812e05598851193449f5684adf036 in 507ms, sequenceid=614, compaction requested=false 2023-04-20 12:53:47,670 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:47,670 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:47,670 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:47,670 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:47,670 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,670 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:47,670 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:47,671 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,672 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:47,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:47,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:47,672 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:47,672 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:47,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:47,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:47,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:47,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:47,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:47,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b44613d[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:47,674 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:47,674 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:47,674 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:47,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:47,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:47,675 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:47,675 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,675 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:47,675 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:47,675 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:47,676 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,676 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:47,676 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:47,676 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:47,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:47,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:47,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:47,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:47,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:47,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:47,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:47,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:47,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:47,680 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:47,680 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:47,680 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:47,680 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:47,680 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:47,681 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:47,681 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:47,681 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:47,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:47,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:47,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:47,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:47,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:47,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:47,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:47,682 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:47,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:47,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:47,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:47,684 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:47,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:47,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:47,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,686 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,686 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:47,686 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:47,686 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:47,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:47,686 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:47,686 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:47,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:47,686 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:47,686 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:47,687 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:47,687 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:47,687 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:47,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:47,687 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:47,687 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:47,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:47,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:47,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on fc3812e05598851193449f5684adf036 2023-04-20 12:53:47,783 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fc3812e05598851193449f5684adf036 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-20 12:53:47,783 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=A 2023-04-20 12:53:47,783 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:53:47,783 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=B 2023-04-20 12:53:47,783 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:53:47,783 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=C 2023-04-20 12:53:47,784 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:53:47,787 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:47,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:47,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:47,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:47,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:47,792 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:47,792 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:47,792 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:47,792 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:47,792 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:47,792 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:47,794 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:47,794 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:47,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:47,794 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,794 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:47,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:47,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:47,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:47,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:47,795 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:47,800 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:47,800 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:47,801 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:47,801 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:47,801 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:47,801 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=653 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/a5a4eea6622a498382819d352fa3bacd 2023-04-20 12:53:47,801 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:47,801 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:47,801 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:47,801 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:47,802 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:47,802 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:47,802 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:47,802 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:47,802 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,802 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:47,802 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:47,809 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,810 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:47,810 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:47,810 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:47,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:47,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:47,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:47,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:47,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:47,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:47,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:47,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3af008fe[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:47,813 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:47,813 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:47,814 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:47,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:47,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:47,814 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:47,814 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:47,814 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:47,814 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,814 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:47,815 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:47,815 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:47,815 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:47,815 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:47,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:47,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:47,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:47,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:47,817 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:47,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32832 deadline: 1681995287817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:47,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:47,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:47,819 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:47,819 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:47,819 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:47,819 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:47,819 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:47,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:47,820 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:47,820 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:47,820 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:47,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:47,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:47,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:47,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:47,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:47,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:47,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:47,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:47,823 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:47,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:47,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:47,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:47,824 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:47,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:47,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,826 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,826 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:47,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:47,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:47,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:47,826 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:47,826 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:47,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:47,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:47,826 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:47,826 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,826 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:47,826 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:47,826 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:47,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:47,826 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:47,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:47,826 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:47,826 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:47,839 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:47,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60994 deadline: 1681995287837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,866 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:47,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32792 deadline: 1681995287865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,866 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:47,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32786 deadline: 1681995287865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,877 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:47,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32782 deadline: 1681995287876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,919 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:47,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32832 deadline: 1681995287918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,927 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:47,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:47,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:47,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:47,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:47,930 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:47,930 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:47,930 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:47,930 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:47,931 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:47,931 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:47,932 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,932 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:47,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:47,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:47,932 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,932 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:47,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:47,932 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:47,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:47,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:47,933 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:47,933 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:47,933 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:47,933 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:47,933 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:47,933 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:47,933 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:47,933 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:47,933 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:47,933 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:47,933 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:47,934 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:47,934 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:47,934 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:47,934 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,934 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:47,934 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:47,935 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,935 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:47,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:47,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:47,935 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:47,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:47,935 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:47,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:47,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:47,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:47,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:47,936 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:47,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e71fd29[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:47,937 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:47,938 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:47,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:47,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:47,938 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:47,938 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,939 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:47,938 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:47,939 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:47,939 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,939 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:47,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:47,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:47,939 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:47,940 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:47,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:47,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:47,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:47,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:47,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:47,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:47,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:47,941 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:47,941 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:47,941 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:47,941 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:47,941 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:47,942 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:47,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:47,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:47,942 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:47,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:47,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:47,942 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:47,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:47,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:47,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:47,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:47,944 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:47,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:47,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:47,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:47,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,945 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:47,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:47,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,946 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:47,946 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:47,946 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:47,946 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:47,946 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:47,946 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:47,946 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:47,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:47,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:47,946 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:47,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:47,946 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:47,946 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:47,946 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:47,947 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:47,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:47,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:48,047 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:48,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:48,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:48,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:48,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:48,064 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:48,064 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:48,065 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:48,065 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:48,065 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,065 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:48,066 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:48,066 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:48,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:48,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:48,066 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,066 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:48,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:48,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:48,067 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:48,078 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:48,078 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:48,078 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:48,078 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:48,078 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:48,078 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:48,078 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:48,078 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:48,078 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:48,079 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:48,079 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:48,079 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:48,079 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:48,079 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,079 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:48,079 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:48,080 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,080 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:48,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:48,081 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:48,081 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:48,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:48,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:48,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:48,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:48,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57d0a0c2[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:48,082 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:48,082 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,083 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,084 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,084 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:48,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,084 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:48,084 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:48,084 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:48,086 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:48,086 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,086 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:48,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:48,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:48,086 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:48,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:48,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:48,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:48,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:48,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:48,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:48,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:48,088 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:48,088 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:48,088 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:48,088 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:48,088 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:48,089 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,089 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,090 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:48,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:48,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:48,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:48,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:48,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:48,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:48,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:48,092 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:48,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:48,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:48,093 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,101 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:48,101 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,101 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:48,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:48,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:48,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:48,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:48,101 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:48,101 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:48,101 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:48,101 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:48,101 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,101 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,101 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:48,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:48,101 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:48,102 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,121 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:48,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32832 deadline: 1681995288120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,202 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:48,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:48,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:48,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:48,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:48,205 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:48,205 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:48,205 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:48,205 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:48,205 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,205 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:48,206 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:48,206 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:48,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:48,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:48,207 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,207 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:48,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:48,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:48,207 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:48,208 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:48,208 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:48,208 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:48,208 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:48,208 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:48,213 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:48,213 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:48,213 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:48,213 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:48,213 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:48,213 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:48,213 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:48,213 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:48,213 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,213 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:48,213 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:48,219 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=653 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/b8fda890af564f37bbd57bb07d32d493 2023-04-20 12:53:48,222 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,222 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:48,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:48,222 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,222 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:48,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:48,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:48,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:48,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:48,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:48,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ff57e8c[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:48,224 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:48,224 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,226 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,226 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:48,226 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:48,226 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:48,227 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,227 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:48,227 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:48,227 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,227 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:48,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:48,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:48,227 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:48,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:48,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:48,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:48,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:48,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:48,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:48,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:48,231 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:48,231 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:48,231 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:48,231 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:48,231 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:48,232 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,233 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:48,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:48,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,233 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:48,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:48,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:48,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:48,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,233 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:48,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:48,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:48,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,234 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:48,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:48,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,237 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:48,237 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:48,237 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:48,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:48,237 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:48,237 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:48,237 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,237 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,237 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,237 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:48,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:48,237 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:48,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:48,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:48,237 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:48,238 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:48,243 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=653 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/89cc93c45b4445dc8875cea52b50ad56 2023-04-20 12:53:48,249 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/a5a4eea6622a498382819d352fa3bacd as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/a5a4eea6622a498382819d352fa3bacd 2023-04-20 12:53:48,254 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/a5a4eea6622a498382819d352fa3bacd, entries=150, sequenceid=653, filesize=11.9 K 2023-04-20 12:53:48,256 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/b8fda890af564f37bbd57bb07d32d493 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/b8fda890af564f37bbd57bb07d32d493 2023-04-20 12:53:48,261 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/b8fda890af564f37bbd57bb07d32d493, entries=150, sequenceid=653, filesize=11.9 K 2023-04-20 12:53:48,262 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/89cc93c45b4445dc8875cea52b50ad56 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/89cc93c45b4445dc8875cea52b50ad56 2023-04-20 12:53:48,268 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/89cc93c45b4445dc8875cea52b50ad56, entries=150, sequenceid=653, filesize=11.9 K 2023-04-20 12:53:48,269 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for fc3812e05598851193449f5684adf036 in 486ms, sequenceid=653, compaction requested=true 2023-04-20 12:53:48,269 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:48,269 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:53:48,269 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:53:48,269 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:53:48,269 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:53:48,269 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:53:48,273 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:53:48,273 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:53:48,273 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): fc3812e05598851193449f5684adf036/B is initiating minor compaction (all files) 2023-04-20 12:53:48,273 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): fc3812e05598851193449f5684adf036/A is initiating minor compaction (all files) 2023-04-20 12:53:48,273 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fc3812e05598851193449f5684adf036/B in TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:48,273 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fc3812e05598851193449f5684adf036/A in TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:48,273 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/d80b97f269514bbb8233513d9e628405, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/35b85ba6987449c284859e423ea2557b, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/a5a4eea6622a498382819d352fa3bacd] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp, totalSize=36.2 K 2023-04-20 12:53:48,273 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/dfcb85b67759440f8d1526b85663e3cc, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/b9bccce7e7b14ca98390dd4f58931c86, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/b8fda890af564f37bbd57bb07d32d493] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp, totalSize=36.2 K 2023-04-20 12:53:48,274 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting d80b97f269514bbb8233513d9e628405, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=603, earliestPutTs=1681995226044 2023-04-20 12:53:48,274 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting dfcb85b67759440f8d1526b85663e3cc, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=603, earliestPutTs=1681995226044 2023-04-20 12:53:48,274 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 35b85ba6987449c284859e423ea2557b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=614, earliestPutTs=1681995226626 2023-04-20 12:53:48,274 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting b9bccce7e7b14ca98390dd4f58931c86, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=614, earliestPutTs=1681995226626 2023-04-20 12:53:48,275 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting a5a4eea6622a498382819d352fa3bacd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=653, earliestPutTs=1681995227233 2023-04-20 12:53:48,275 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting b8fda890af564f37bbd57bb07d32d493, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=653, earliestPutTs=1681995227233 2023-04-20 12:53:48,282 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): fc3812e05598851193449f5684adf036#B#compaction#56 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:53:48,283 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): fc3812e05598851193449f5684adf036#A#compaction#57 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:53:48,338 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:48,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:48,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:48,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:48,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:48,341 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:48,341 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:48,341 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:48,341 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:48,341 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,341 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:48,342 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,342 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:48,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:48,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:48,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:48,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:48,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:48,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on fc3812e05598851193449f5684adf036 2023-04-20 12:53:48,344 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:48,344 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fc3812e05598851193449f5684adf036 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-20 12:53:48,345 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=A 2023-04-20 12:53:48,345 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:53:48,345 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=B 2023-04-20 12:53:48,345 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:53:48,345 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=C 2023-04-20 12:53:48,345 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:53:48,348 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,348 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:48,354 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:48,354 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:48,354 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:48,354 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:48,354 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:48,354 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:48,354 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:48,354 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:48,355 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:48,355 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:48,355 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:48,355 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:48,355 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:48,355 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,355 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:48,355 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:48,358 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,358 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:48,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:48,358 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,358 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:48,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:48,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:48,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:48,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:48,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:48,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f82674[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:48,360 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:48,360 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,362 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,362 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:48,362 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:48,362 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:48,363 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:48,363 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:48,363 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:48,363 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,364 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:48,364 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:48,364 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:48,364 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:48,364 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:48,364 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:48,364 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:48,366 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=664 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/6b5d4f0595a9498c87a9a25c6b74ac78 2023-04-20 12:53:48,367 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,367 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,367 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:48,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:48,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:48,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:48,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:48,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:48,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,368 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:48,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:48,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:48,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:48,370 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:48,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,373 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:48,374 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:48,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:48,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:48,375 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,375 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:48,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:48,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:48,375 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:48,376 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,376 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,376 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,376 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,375 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:48,376 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:48,376 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:48,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:48,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:48,384 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=664 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/42ceec74dd204cf2a28b6ac3054967b8 2023-04-20 12:53:48,385 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:48,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32786 deadline: 1681995288380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,386 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:48,386 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681995228374, max timestamp=1681995228384Num uniques 859; 2023-04-20 12:53:48,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32792 deadline: 1681995288384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,386 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681995228374, max timestamp=1681995228384Num uniques 859; 2023-04-20 12:53:48,386 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-20 12:53:48,386 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681995228374, max timestamp=1681995228384Num uniques 859; 2023-04-20 12:53:48,387 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-20 12:53:48,386 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-20 12:53:48,398 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=664 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/a4780354b5ad4a9d93e998a640badd1c 2023-04-20 12:53:48,403 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/6b5d4f0595a9498c87a9a25c6b74ac78 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/6b5d4f0595a9498c87a9a25c6b74ac78 2023-04-20 12:53:48,409 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/6b5d4f0595a9498c87a9a25c6b74ac78, entries=200, sequenceid=664, filesize=14.3 K 2023-04-20 12:53:48,410 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/42ceec74dd204cf2a28b6ac3054967b8 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/42ceec74dd204cf2a28b6ac3054967b8 2023-04-20 12:53:48,413 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:48,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32782 deadline: 1681995288409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,416 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/42ceec74dd204cf2a28b6ac3054967b8, entries=150, sequenceid=664, filesize=11.9 K 2023-04-20 12:53:48,416 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:48,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60994 deadline: 1681995288412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,418 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/a4780354b5ad4a9d93e998a640badd1c as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/a4780354b5ad4a9d93e998a640badd1c 2023-04-20 12:53:48,424 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/a4780354b5ad4a9d93e998a640badd1c, entries=150, sequenceid=664, filesize=11.9 K 2023-04-20 12:53:48,425 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for fc3812e05598851193449f5684adf036 in 81ms, sequenceid=664, compaction requested=true 2023-04-20 12:53:48,425 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:48,425 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-20 12:53:48,425 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-20 12:53:48,425 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-04-20 12:53:48,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on fc3812e05598851193449f5684adf036 2023-04-20 12:53:48,431 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fc3812e05598851193449f5684adf036 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-20 12:53:48,431 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=A 2023-04-20 12:53:48,431 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:53:48,431 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=B 2023-04-20 12:53:48,431 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:53:48,431 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=C 2023-04-20 12:53:48,431 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:53:48,458 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:48,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32832 deadline: 1681995288457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,476 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:48,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:48,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:48,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:48,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:48,479 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:48,479 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:48,479 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:48,479 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:48,480 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,480 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:48,480 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:48,480 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:48,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:48,481 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,481 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:48,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:48,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:48,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:48,481 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:48,482 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:48,482 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:48,482 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:48,482 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:48,482 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:48,482 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:48,482 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:48,482 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:48,482 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:48,482 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:48,482 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:48,482 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:48,482 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:48,482 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,482 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:48,482 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:48,485 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,485 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:48,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:48,486 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,486 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:48,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:48,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:48,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:48,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:48,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:48,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e10b57[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:48,487 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:48,487 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,487 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,488 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:48,488 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:48,488 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:48,488 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,488 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:48,488 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,488 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:48,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:48,489 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:48,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:48,489 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:48,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:48,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:48,489 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:48,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:48,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32792 deadline: 1681995288487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:48,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:48,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:48,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:48,491 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:48,491 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:48,491 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:48,491 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:48,491 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:48,492 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,492 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:48,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,492 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:48,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:48,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:48,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:48,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:48,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,493 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:48,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32786 deadline: 1681995288492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,494 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:48,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:48,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:48,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,495 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:48,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:48,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,497 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,497 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:48,497 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:48,497 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:48,497 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:48,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:48,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:48,497 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:48,497 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:48,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:48,497 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,498 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,498 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:48,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:48,498 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:48,498 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:48,516 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:48,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32782 deadline: 1681995288515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,518 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:48,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60994 deadline: 1681995288517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,561 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:48,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32832 deadline: 1681995288560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,598 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:48,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:48,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:48,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:48,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:48,601 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:48,601 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:48,601 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:48,601 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:48,606 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,606 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:48,607 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:48,607 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:48,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:48,607 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,608 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:48,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:48,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:48,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:48,608 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:48,609 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:48,609 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:48,609 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:48,609 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:48,609 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:48,610 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:48,610 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:48,610 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:48,610 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:48,610 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:48,610 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:48,610 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:48,610 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:48,610 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,610 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:48,610 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:48,611 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,611 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:48,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:48,612 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,612 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:48,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:48,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:48,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:48,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:48,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:48,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56657db9[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:48,614 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:48,614 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,614 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,615 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:48,615 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:48,615 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:48,615 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,615 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:48,617 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:48,617 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,617 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:48,618 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:48,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:48,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:48,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:48,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:48,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:48,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:48,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:48,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:48,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:48,623 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:48,623 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:48,623 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:48,623 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:48,623 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:48,631 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,631 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,631 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:48,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:48,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:48,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:48,631 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:48,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:48,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:48,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:48,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:48,636 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:48,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:48,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,655 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:48,655 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:48,655 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:48,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:48,655 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:48,655 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:48,655 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:48,655 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,655 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,655 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,655 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:48,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:48,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:48,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:48,655 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:48,655 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:48,655 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:48,693 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:48,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32792 deadline: 1681995288692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,696 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:48,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32786 deadline: 1681995288696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,698 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/d8642be750364a77a8c5e8afe1a9b82f as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/d8642be750364a77a8c5e8afe1a9b82f 2023-04-20 12:53:48,707 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fc3812e05598851193449f5684adf036/B of fc3812e05598851193449f5684adf036 into d8642be750364a77a8c5e8afe1a9b82f(size=12.5 K), total size for store is 24.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:53:48,707 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:48,707 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036., storeName=fc3812e05598851193449f5684adf036/B, priority=13, startTime=1681995228269; duration=0sec 2023-04-20 12:53:48,708 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-04-20 12:53:48,708 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 3 compacting, 1 eligible, 16 blocking 2023-04-20 12:53:48,709 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-20 12:53:48,709 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-20 12:53:48,709 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. because compaction request was cancelled 2023-04-20 12:53:48,709 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/cb98bb476a524fd2b78738a30d5756c3 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/cb98bb476a524fd2b78738a30d5756c3 2023-04-20 12:53:48,709 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 12:53:48,711 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49256 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 12:53:48,711 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): fc3812e05598851193449f5684adf036/C is initiating minor compaction (all files) 2023-04-20 12:53:48,712 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fc3812e05598851193449f5684adf036/C in TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:48,712 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/c638e4cbfb4141f281f070ea1885ae0f, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/10c83a7919ff438fb1e051e13ff01a1d, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/89cc93c45b4445dc8875cea52b50ad56, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/a4780354b5ad4a9d93e998a640badd1c] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp, totalSize=48.1 K 2023-04-20 12:53:48,712 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting c638e4cbfb4141f281f070ea1885ae0f, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=603, earliestPutTs=1681995226044 2023-04-20 12:53:48,713 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 10c83a7919ff438fb1e051e13ff01a1d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=614, earliestPutTs=1681995226626 2023-04-20 12:53:48,713 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 89cc93c45b4445dc8875cea52b50ad56, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=653, earliestPutTs=1681995227233 2023-04-20 12:53:48,714 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting a4780354b5ad4a9d93e998a640badd1c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=664, earliestPutTs=1681995227804 2023-04-20 12:53:48,718 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fc3812e05598851193449f5684adf036/A of fc3812e05598851193449f5684adf036 into cb98bb476a524fd2b78738a30d5756c3(size=12.5 K), total size for store is 26.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:53:48,718 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:48,718 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036., storeName=fc3812e05598851193449f5684adf036/A, priority=13, startTime=1681995228269; duration=0sec 2023-04-20 12:53:48,718 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-20 12:53:48,718 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 4 compacting, 0 eligible, 16 blocking 2023-04-20 12:53:48,718 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-20 12:53:48,718 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-20 12:53:48,719 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. because compaction request was cancelled 2023-04-20 12:53:48,719 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-20 12:53:48,720 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-20 12:53:48,720 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-20 12:53:48,720 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. because compaction request was cancelled 2023-04-20 12:53:48,724 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:48,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32782 deadline: 1681995288720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,725 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:48,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60994 deadline: 1681995288720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,727 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): fc3812e05598851193449f5684adf036#C#compaction#62 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:53:48,739 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/ced07e9101ae4c429c0b773207715999 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/ced07e9101ae4c429c0b773207715999 2023-04-20 12:53:48,745 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in fc3812e05598851193449f5684adf036/C of fc3812e05598851193449f5684adf036 into ced07e9101ae4c429c0b773207715999(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:53:48,745 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:48,745 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036., storeName=fc3812e05598851193449f5684adf036/C, priority=12, startTime=1681995228425; duration=0sec 2023-04-20 12:53:48,745 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:53:48,756 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:48,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:48,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:48,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:48,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:48,759 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:48,759 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:48,759 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:48,759 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:48,759 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,759 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:48,760 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:48,760 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:48,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:48,760 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,760 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:48,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:48,760 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:48,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:48,761 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:48,761 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:48,761 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:48,761 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:48,761 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:48,761 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:48,761 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:48,761 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:48,761 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:48,761 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:48,761 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:48,762 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:48,762 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:48,762 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:48,762 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,762 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:48,762 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:48,763 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:48,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:48,763 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:48,763 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,764 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:48,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:48,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:48,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:48,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:48,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@427078e7[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:48,764 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:48,765 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,765 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,765 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:48,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32832 deadline: 1681995288762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,765 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:48,765 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:48,765 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:48,766 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,766 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:48,766 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,766 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:48,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:48,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:48,766 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:48,766 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:48,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:48,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:48,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:48,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:48,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:48,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:48,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:48,768 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:48,769 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:48,769 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:48,769 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:48,769 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:48,770 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,770 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,770 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:48,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:48,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:48,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:48,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:48,770 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:48,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:48,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:48,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:48,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,771 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:48,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:48,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,773 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,773 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:48,773 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:48,773 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:48,773 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,773 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,773 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,773 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:48,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:48,773 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:48,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:48,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:48,773 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:48,773 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:48,773 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:48,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:48,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:48,845 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=699 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/9e248d37c9fd42b6ad13bd2faad13cec 2023-04-20 12:53:48,873 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:48,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:48,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:48,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:48,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:48,877 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:48,877 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:48,877 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:48,877 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:48,877 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,877 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:48,878 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,878 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:48,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:48,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:48,878 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,878 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:48,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:48,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:48,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:48,879 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:48,879 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:48,879 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:48,879 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:48,879 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:48,879 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:48,879 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:48,879 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:48,879 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:48,880 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:48,880 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:48,880 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:48,880 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:48,880 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:48,880 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,880 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:48,880 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:48,881 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,881 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:48,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:48,881 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,881 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:48,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:48,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:48,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:48,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:48,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:48,882 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:48,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67f2d5b[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:48,882 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,883 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,883 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:48,883 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:48,883 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,883 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:48,883 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:48,884 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,884 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:48,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:48,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:48,884 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:48,884 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:48,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:48,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:48,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:48,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:48,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:48,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:48,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:48,886 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:48,886 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:48,886 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:48,886 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:48,886 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:48,887 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,887 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:48,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:48,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,887 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:48,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:48,888 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:48,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:48,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:48,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:48,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:48,889 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:48,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:48,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,890 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,890 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:48,890 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:48,890 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:48,891 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,891 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:48,891 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:48,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:48,891 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:48,891 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:48,891 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:48,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:48,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:48,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:48,891 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:48,891 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:48,891 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:48,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:48,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:49,004 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:49,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:49,005 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:49,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32786 deadline: 1681995289005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:49,007 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:49,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32792 deadline: 1681995289005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:49,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:49,007 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:49,007 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:49,007 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:49,007 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:49,008 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:49,008 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:49,008 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:49,008 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:49,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:49,009 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,009 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:49,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:49,009 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:49,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:49,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:49,009 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:49,011 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:49,011 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:49,011 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:49,011 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:49,011 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:49,011 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:49,011 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:49,011 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:49,011 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:49,011 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:49,011 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:49,011 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:49,011 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,011 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:49,011 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:49,012 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,012 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:49,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:49,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:49,013 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:49,013 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:49,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:49,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:49,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:49,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:49,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:49,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a482570[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:49,014 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:49,014 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:49,014 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:49,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:49,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:49,015 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:49,015 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:49,015 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:49,015 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,015 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:49,015 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,016 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:49,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:49,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:49,016 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:49,016 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:49,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:49,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:49,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:49,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:49,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:49,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:49,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:49,018 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:49,018 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:49,018 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:49,018 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:49,018 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:49,020 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:49,020 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:49,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:49,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:49,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:49,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:49,020 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:49,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:49,020 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:49,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:49,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:49,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:49,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:49,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:49,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:49,022 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:49,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:49,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,024 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,024 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:49,024 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:49,024 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,024 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:49,024 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:49,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:49,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:49,024 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:49,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:49,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:49,024 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:49,024 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:49,024 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:49,024 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:49,024 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:49,024 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:49,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:49,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:49,026 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:49,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32782 deadline: 1681995289026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,027 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:49,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60994 deadline: 1681995289027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,069 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:49,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32832 deadline: 1681995289068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,124 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:49,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:49,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:49,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:49,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:49,128 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:49,128 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:49,128 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:49,128 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:49,129 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:49,129 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:49,130 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:49,130 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:49,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:49,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:49,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:49,131 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,131 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:49,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:49,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:49,131 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:49,131 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:49,131 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:49,131 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:49,131 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:49,131 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:49,131 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:49,131 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:49,131 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:49,131 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:49,131 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:49,131 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:49,131 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:49,131 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:49,132 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,132 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:49,132 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:49,132 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,132 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:49,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:49,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:49,133 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:49,133 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:49,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:49,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:49,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:49,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:49,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:49,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@773d0935[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:49,134 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:49,134 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:49,135 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:49,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:49,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:49,135 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:49,135 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:49,135 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,135 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:49,135 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:49,136 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,136 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:49,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:49,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:49,136 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:49,136 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:49,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:49,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:49,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:49,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:49,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:49,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:49,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:49,138 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:49,138 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:49,138 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:49,138 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:49,138 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:49,139 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:49,139 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:49,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:49,139 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:49,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:49,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:49,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:49,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:49,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:49,139 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:49,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:49,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:49,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:49,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:49,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:49,140 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:49,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:49,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,144 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:49,144 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:49,144 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:49,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:49,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:49,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:49,144 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:49,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:49,144 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:49,144 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:49,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:49,144 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:49,144 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:49,145 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,145 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:49,145 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:49,145 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:49,245 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:49,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:49,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:49,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:49,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:49,248 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:49,248 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:49,248 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:49,248 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:49,248 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:49,248 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:49,249 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:49,249 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:49,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:49,249 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:49,249 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:49,250 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:49,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:49,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:49,250 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:49,250 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:49,250 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:49,250 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:49,250 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:49,250 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:49,251 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:49,251 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:49,251 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:49,251 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:49,251 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:49,251 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:49,251 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:49,251 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:49,251 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,251 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:49,251 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:49,252 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,252 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:49,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:49,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:49,252 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:49,252 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:49,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:49,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:49,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:49,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:49,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:49,253 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:49,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36cbf075[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:49,253 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:49,254 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:49,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:49,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:49,254 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:49,254 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:49,254 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:49,254 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,255 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:49,255 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,255 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:49,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:49,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:49,255 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:49,255 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:49,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:49,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:49,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:49,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:49,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:49,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:49,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:49,257 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:49,257 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:49,257 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:49,257 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:49,257 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:49,258 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=699 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/ad789420b4104b96a940eab738cb097d 2023-04-20 12:53:49,259 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:49,259 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:49,259 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:49,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:49,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:49,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:49,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:49,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:49,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:49,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:49,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:49,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:49,261 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:49,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:49,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:49,263 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:49,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:49,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,271 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:49,271 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:49,271 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:49,271 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,271 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:49,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:49,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:49,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:49,271 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:49,272 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:49,272 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,272 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:49,272 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:49,272 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:49,272 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:49,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:49,272 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:49,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:49,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:49,283 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=699 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/2fc2928bc712431c8de9a78335e480b6 2023-04-20 12:53:49,288 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/9e248d37c9fd42b6ad13bd2faad13cec as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/9e248d37c9fd42b6ad13bd2faad13cec 2023-04-20 12:53:49,292 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/9e248d37c9fd42b6ad13bd2faad13cec, entries=150, sequenceid=699, filesize=11.9 K 2023-04-20 12:53:49,293 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/ad789420b4104b96a940eab738cb097d as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/ad789420b4104b96a940eab738cb097d 2023-04-20 12:53:49,306 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/ad789420b4104b96a940eab738cb097d, entries=150, sequenceid=699, filesize=11.9 K 2023-04-20 12:53:49,307 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/2fc2928bc712431c8de9a78335e480b6 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/2fc2928bc712431c8de9a78335e480b6 2023-04-20 12:53:49,313 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/2fc2928bc712431c8de9a78335e480b6, entries=150, sequenceid=699, filesize=11.9 K 2023-04-20 12:53:49,313 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for fc3812e05598851193449f5684adf036 in 882ms, sequenceid=699, compaction requested=true 2023-04-20 12:53:49,314 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:49,314 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:53:49,314 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:53:49,314 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:53:49,314 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:53:49,314 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:53:49,315 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39646 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:53:49,315 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): fc3812e05598851193449f5684adf036/A is initiating minor compaction (all files) 2023-04-20 12:53:49,315 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:53:49,315 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fc3812e05598851193449f5684adf036/A in TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:49,315 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): fc3812e05598851193449f5684adf036/B is initiating minor compaction (all files) 2023-04-20 12:53:49,315 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fc3812e05598851193449f5684adf036/B in TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:49,315 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/cb98bb476a524fd2b78738a30d5756c3, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/6b5d4f0595a9498c87a9a25c6b74ac78, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/9e248d37c9fd42b6ad13bd2faad13cec] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp, totalSize=38.7 K 2023-04-20 12:53:49,315 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/d8642be750364a77a8c5e8afe1a9b82f, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/42ceec74dd204cf2a28b6ac3054967b8, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/ad789420b4104b96a940eab738cb097d] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp, totalSize=36.3 K 2023-04-20 12:53:49,316 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting cb98bb476a524fd2b78738a30d5756c3, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=653, earliestPutTs=1681995227233 2023-04-20 12:53:49,316 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting d8642be750364a77a8c5e8afe1a9b82f, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=653, earliestPutTs=1681995227233 2023-04-20 12:53:49,316 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 6b5d4f0595a9498c87a9a25c6b74ac78, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=664, earliestPutTs=1681995227804 2023-04-20 12:53:49,317 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 42ceec74dd204cf2a28b6ac3054967b8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=664, earliestPutTs=1681995227804 2023-04-20 12:53:49,317 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 9e248d37c9fd42b6ad13bd2faad13cec, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=699, earliestPutTs=1681995228394 2023-04-20 12:53:49,317 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting ad789420b4104b96a940eab738cb097d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=699, earliestPutTs=1681995228394 2023-04-20 12:53:49,326 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): fc3812e05598851193449f5684adf036#B#compaction#65 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:53:49,330 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): fc3812e05598851193449f5684adf036#A#compaction#66 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:53:49,343 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/a214f6ecfda44a83a3bffc62c428aca7 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/a214f6ecfda44a83a3bffc62c428aca7 2023-04-20 12:53:49,344 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/6759b824d89340a09fd4a2208ddcedb9 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/6759b824d89340a09fd4a2208ddcedb9 2023-04-20 12:53:49,351 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fc3812e05598851193449f5684adf036/B of fc3812e05598851193449f5684adf036 into a214f6ecfda44a83a3bffc62c428aca7(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:53:49,351 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:49,351 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036., storeName=fc3812e05598851193449f5684adf036/B, priority=13, startTime=1681995229314; duration=0sec 2023-04-20 12:53:49,351 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:53:49,351 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-20 12:53:49,352 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-20 12:53:49,352 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-20 12:53:49,352 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. because compaction request was cancelled 2023-04-20 12:53:49,353 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fc3812e05598851193449f5684adf036/A of fc3812e05598851193449f5684adf036 into 6759b824d89340a09fd4a2208ddcedb9(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:53:49,353 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:49,353 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036., storeName=fc3812e05598851193449f5684adf036/A, priority=13, startTime=1681995229314; duration=0sec 2023-04-20 12:53:49,353 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:53:49,372 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:49,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:49,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:49,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:49,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:49,375 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:49,375 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:49,375 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:49,375 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:49,375 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:49,375 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:49,376 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:49,376 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:49,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:49,376 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,376 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:49,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:49,377 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:49,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:49,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:49,377 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:49,377 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:49,377 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:49,377 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:49,377 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:49,377 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:49,378 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:49,378 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:49,378 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:49,378 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:49,378 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fc3812e05598851193449f5684adf036 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-20 12:53:49,378 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=A 2023-04-20 12:53:49,378 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:53:49,378 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=B 2023-04-20 12:53:49,378 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:53:49,378 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=C 2023-04-20 12:53:49,378 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:53:49,386 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=714 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/7f9286b9a9db424ebda6519b8c85ba84 2023-04-20 12:53:49,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on fc3812e05598851193449f5684adf036 2023-04-20 12:53:49,508 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:49,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16434, totalHeapSize=37576, min timestamp=1681995229514, max timestamp=1681995229528Num uniques 859; 2023-04-20 12:53:49,530 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=16434, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-20 12:53:49,531 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16434, totalHeapSize=37576, min timestamp=1681995229514, max timestamp=1681995229528Num uniques 859; 2023-04-20 12:53:49,531 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=16434, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-20 12:53:49,531 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16434, totalHeapSize=37576, min timestamp=1681995229514, max timestamp=1681995229528Num uniques 859; 2023-04-20 12:53:49,531 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=16434, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-20 12:53:49,550 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:49,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32786 deadline: 1681995289546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,551 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:49,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32792 deadline: 1681995289546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,551 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:49,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32782 deadline: 1681995289547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,551 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:49,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60994 deadline: 1681995289547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,572 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:49,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32832 deadline: 1681995289571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,653 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:49,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32792 deadline: 1681995289652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,654 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:49,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32786 deadline: 1681995289652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,654 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:49,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32782 deadline: 1681995289652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,654 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:49,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60994 deadline: 1681995289652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,797 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=714 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/23d0442755e640599fd3ff90ce861c42 2023-04-20 12:53:49,856 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:49,856 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:49,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32786 deadline: 1681995289855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32792 deadline: 1681995289855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,857 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:49,857 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:49,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60994 deadline: 1681995289855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:49,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32782 deadline: 1681995289856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:50,159 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:50,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32792 deadline: 1681995290158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:50,162 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:50,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32786 deadline: 1681995290158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:50,163 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:50,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32782 deadline: 1681995290159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:50,163 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:50,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60994 deadline: 1681995290160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:50,210 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=714 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/69e6ebe7b57e4062b788b93869f71b2f 2023-04-20 12:53:50,215 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/7f9286b9a9db424ebda6519b8c85ba84 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/7f9286b9a9db424ebda6519b8c85ba84 2023-04-20 12:53:50,220 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/7f9286b9a9db424ebda6519b8c85ba84, entries=150, sequenceid=714, filesize=11.9 K 2023-04-20 12:53:50,221 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/23d0442755e640599fd3ff90ce861c42 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/23d0442755e640599fd3ff90ce861c42 2023-04-20 12:53:50,226 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/23d0442755e640599fd3ff90ce861c42, entries=150, sequenceid=714, filesize=11.9 K 2023-04-20 12:53:50,227 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/69e6ebe7b57e4062b788b93869f71b2f as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/69e6ebe7b57e4062b788b93869f71b2f 2023-04-20 12:53:50,232 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/69e6ebe7b57e4062b788b93869f71b2f, entries=150, sequenceid=714, filesize=11.9 K 2023-04-20 12:53:50,233 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=147.60 KB/151140 for fc3812e05598851193449f5684adf036 in 855ms, sequenceid=714, compaction requested=true 2023-04-20 12:53:50,233 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:50,233 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:50,233 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:50,233 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:50,233 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:50,233 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:50,233 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:50,234 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:50,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:50,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:50,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:50,235 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:50,235 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:50,235 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:50,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:50,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:50,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:50,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:50,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:50,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:50,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49211602[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:50,236 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:50,236 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:50,237 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:50,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:50,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:50,237 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:50,237 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:50,237 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:50,237 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:50,237 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:50,238 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:50,238 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:50,238 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:50,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:50,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:50,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:50,238 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:50,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:50,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:50,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:50,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:50,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:50,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:50,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:50,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:50,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:50,240 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:50,240 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:50,240 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:50,240 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:50,240 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:50,241 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:50,241 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:50,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:50,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:50,241 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:50,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:50,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:50,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:50,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:50,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:50,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:50,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:50,241 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:50,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:50,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:50,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:50,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:50,243 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:50,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:50,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:50,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:50,245 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:50,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:50,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:50,245 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:50,245 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:50,245 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:50,245 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:50,245 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:50,245 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:50,245 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:50,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:50,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:50,245 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:50,245 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:50,245 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:50,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:50,245 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:50,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:50,346 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:50,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:50,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:50,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:50,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:50,349 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:50,349 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:50,349 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:50,349 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:50,349 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:50,349 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:50,350 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:50,350 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:50,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:50,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:50,350 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:50,350 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:50,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:50,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:50,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:50,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:50,359 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:50,359 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:50,359 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:50,359 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:50,359 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:50,359 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:50,361 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:50,361 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:50,361 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:50,361 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:50,361 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fc3812e05598851193449f5684adf036 3/3 column families, dataSize=147.60 KB heapSize=406.08 KB 2023-04-20 12:53:50,361 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=A 2023-04-20 12:53:50,361 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:53:50,361 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=B 2023-04-20 12:53:50,361 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:53:50,361 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=C 2023-04-20 12:53:50,362 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:53:50,380 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=749 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/01426621474b40eab00989d3de456745 2023-04-20 12:53:50,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on fc3812e05598851193449f5684adf036 2023-04-20 12:53:50,583 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:50,599 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:50,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32832 deadline: 1681995290598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:50,665 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:50,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32782 deadline: 1681995290664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:50,665 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:50,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32792 deadline: 1681995290664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:50,665 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:50,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32786 deadline: 1681995290664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:50,666 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:50,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60994 deadline: 1681995290665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:50,701 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:50,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32832 deadline: 1681995290700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:50,807 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=749 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/5bb9db7c1cd842ef87f1cfed6e7ef1f0 2023-04-20 12:53:50,904 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:50,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32832 deadline: 1681995290903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:51,208 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:51,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32832 deadline: 1681995291207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:51,222 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=749 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/ac914b06475449d49bacd87ca89646ea 2023-04-20 12:53:51,228 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/01426621474b40eab00989d3de456745 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/01426621474b40eab00989d3de456745 2023-04-20 12:53:51,234 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/01426621474b40eab00989d3de456745, entries=150, sequenceid=749, filesize=11.9 K 2023-04-20 12:53:51,235 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/5bb9db7c1cd842ef87f1cfed6e7ef1f0 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/5bb9db7c1cd842ef87f1cfed6e7ef1f0 2023-04-20 12:53:51,240 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/5bb9db7c1cd842ef87f1cfed6e7ef1f0, entries=150, sequenceid=749, filesize=11.9 K 2023-04-20 12:53:51,241 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/ac914b06475449d49bacd87ca89646ea as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/ac914b06475449d49bacd87ca89646ea 2023-04-20 12:53:51,246 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/ac914b06475449d49bacd87ca89646ea, entries=150, sequenceid=749, filesize=11.9 K 2023-04-20 12:53:51,247 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~406.03 KB/415776, currentSize=40.25 KB/41220 for fc3812e05598851193449f5684adf036 in 886ms, sequenceid=749, compaction requested=true 2023-04-20 12:53:51,247 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:51,247 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:51,247 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:51,247 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:51,247 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:51,247 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:51,247 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:51,248 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:51,248 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:51,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:51,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:51,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:51,248 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:51,248 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:51,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:51,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:51,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:51,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:51,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:51,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:51,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@289c9f3[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:51,249 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:51,249 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:51,250 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:51,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:51,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:51,250 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:51,250 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:51,250 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:51,250 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:51,250 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:51,251 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:51,251 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:51,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:51,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:51,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:51,251 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:51,251 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:51,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:51,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:51,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:51,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:51,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:51,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:51,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:51,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:51,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:51,253 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:51,253 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:51,253 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:51,253 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:51,253 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:51,254 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:51,254 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:51,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:51,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:51,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:51,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:51,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:51,254 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:51,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:51,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:51,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:51,255 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:51,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:51,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:51,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:51,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:51,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:51,256 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:51,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:51,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:51,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:51,258 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:51,258 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:51,258 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:51,258 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:51,258 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:51,258 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:51,258 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:51,258 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:51,258 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:51,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:51,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:51,258 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:51,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:51,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:51,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:51,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:51,259 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:51,259 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:51,359 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:51,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:51,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:51,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:51,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:51,366 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:51,366 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:51,366 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:51,366 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:51,367 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:51,367 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:51,367 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:51,367 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:51,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:51,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:51,368 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:51,368 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:51,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:51,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:51,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:51,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:51,369 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:51,374 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:51,374 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:51,374 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:51,374 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:51,374 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:51,382 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:51,382 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:51,382 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:51,382 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:51,382 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fc3812e05598851193449f5684adf036 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-20 12:53:51,382 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=A 2023-04-20 12:53:51,383 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:53:51,383 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=B 2023-04-20 12:53:51,383 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:53:51,383 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=C 2023-04-20 12:53:51,383 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:53:51,405 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=758 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/3a6988e4560d47e6829169d3dedd7569 2023-04-20 12:53:51,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on fc3812e05598851193449f5684adf036 2023-04-20 12:53:51,669 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:51,685 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681995231676, max timestamp=1681995231684Num uniques 859; 2023-04-20 12:53:51,685 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-20 12:53:51,686 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681995231676, max timestamp=1681995231684Num uniques 859; 2023-04-20 12:53:51,686 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-20 12:53:51,686 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681995231676, max timestamp=1681995231684Num uniques 859; 2023-04-20 12:53:51,686 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-20 12:53:51,697 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:51,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32782 deadline: 1681995291695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:51,698 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:51,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32792 deadline: 1681995291696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:51,699 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:51,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60994 deadline: 1681995291696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:51,699 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:51,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32786 deadline: 1681995291697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:51,714 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:51,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32832 deadline: 1681995291712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:51,800 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:51,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32782 deadline: 1681995291798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:51,801 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:51,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32792 deadline: 1681995291798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:51,801 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:51,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60994 deadline: 1681995291800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:51,801 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:51,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32786 deadline: 1681995291800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:51,820 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=758 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/f59fcb4473e642269925310c4701c745 2023-04-20 12:53:52,002 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:52,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32792 deadline: 1681995292002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,002 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:52,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60994 deadline: 1681995292002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,003 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:52,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32786 deadline: 1681995292002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,003 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:52,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32782 deadline: 1681995292002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,232 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=758 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/2118ba1f99a249a58d3f13969814c1cd 2023-04-20 12:53:52,239 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/3a6988e4560d47e6829169d3dedd7569 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/3a6988e4560d47e6829169d3dedd7569 2023-04-20 12:53:52,244 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/3a6988e4560d47e6829169d3dedd7569, entries=150, sequenceid=758, filesize=11.9 K 2023-04-20 12:53:52,245 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/f59fcb4473e642269925310c4701c745 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/f59fcb4473e642269925310c4701c745 2023-04-20 12:53:52,250 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/f59fcb4473e642269925310c4701c745, entries=150, sequenceid=758, filesize=11.9 K 2023-04-20 12:53:52,251 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/2118ba1f99a249a58d3f13969814c1cd as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/2118ba1f99a249a58d3f13969814c1cd 2023-04-20 12:53:52,256 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/2118ba1f99a249a58d3f13969814c1cd, entries=150, sequenceid=758, filesize=11.9 K 2023-04-20 12:53:52,257 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=147.60 KB/151140 for fc3812e05598851193449f5684adf036 in 875ms, sequenceid=758, compaction requested=true 2023-04-20 12:53:52,257 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:52,257 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:52,257 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:52,257 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:52,258 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,258 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:52,258 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:52,259 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,259 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:52,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:52,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:52,259 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,259 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:52,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:52,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:52,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:52,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:52,260 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:52,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d621c36[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:52,260 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,261 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,261 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:52,261 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:52,261 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:52,261 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,261 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:52,262 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:52,262 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:52,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:52,262 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:52,262 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:52,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:52,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:52,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:52,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:52,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:52,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:52,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:52,264 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:52,264 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:52,264 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:52,264 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:52,264 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:52,265 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,265 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,265 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:52,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:52,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:52,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:52,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:52,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:52,267 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:52,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:52,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:52,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,268 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:52,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:52,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,269 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,269 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:52,269 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:52,269 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:52,269 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:52,269 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:52,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:52,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:52,269 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:52,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:52,270 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:52,269 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,270 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:52,270 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,270 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:52,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:52,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on fc3812e05598851193449f5684adf036 2023-04-20 12:53:52,305 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fc3812e05598851193449f5684adf036 3/3 column families, dataSize=154.31 KB heapSize=426 KB 2023-04-20 12:53:52,306 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=A 2023-04-20 12:53:52,306 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:53:52,306 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=B 2023-04-20 12:53:52,306 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:53:52,306 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=C 2023-04-20 12:53:52,306 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:53:52,310 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:52,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32786 deadline: 1681995292307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,310 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:52,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32782 deadline: 1681995292307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,312 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:52,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32792 deadline: 1681995292310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,313 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:52,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60994 deadline: 1681995292310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,370 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:52,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:52,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:52,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:52,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:52,373 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:52,373 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:52,373 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:52,373 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:52,374 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,374 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:52,375 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:52,375 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:52,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:52,375 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,375 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:52,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:52,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:52,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:52,376 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:52,376 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:52,376 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:52,376 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:52,376 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:52,376 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:52,376 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:52,376 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:52,376 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:52,376 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:52,376 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:52,376 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:52,377 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:52,377 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:52,377 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,377 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:52,377 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:52,378 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,378 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:52,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:52,378 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,378 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:52,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:52,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:52,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:52,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:52,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:52,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74431fb7[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:52,379 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:52,379 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,380 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,380 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:52,380 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:52,380 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:52,380 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,380 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:52,381 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:52,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:52,381 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:52,381 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:52,381 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:52,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:52,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:52,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:52,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:52,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:52,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:52,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:52,383 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:52,383 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:52,383 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:52,383 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:52,383 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:52,384 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,385 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:52,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,385 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:52,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:52,385 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:52,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:52,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:52,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:52,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:52,386 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:52,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:52,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:52,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,388 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,388 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:52,388 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:52,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:52,389 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:52,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:52,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:52,389 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:52,389 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,389 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,389 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,389 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,389 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:52,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:52,389 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:52,389 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:52,389 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:52,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:52,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:52,411 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:52,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32786 deadline: 1681995292411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,412 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:52,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32782 deadline: 1681995292411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,415 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:52,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32792 deadline: 1681995292413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,415 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:52,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60994 deadline: 1681995292413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,489 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:52,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:52,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:52,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:52,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:52,492 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:52,492 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:52,492 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:52,492 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:52,493 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,493 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:52,494 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,494 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:52,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:52,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:52,494 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,494 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:52,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:52,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:52,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:52,495 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:52,495 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:52,495 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:52,495 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:52,495 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:52,495 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:52,495 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:52,495 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:52,495 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:52,495 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:52,495 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:52,496 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:52,496 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:52,496 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:52,496 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,496 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:52,496 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:52,497 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,497 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:52,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:52,497 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,497 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:52,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:52,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:52,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:52,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:52,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:52,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3513963b[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:52,498 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:52,498 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,499 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,499 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:52,499 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:52,499 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:52,499 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,499 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:52,500 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,500 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:52,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:52,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:52,500 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:52,500 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:52,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:52,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:52,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:52,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:52,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:52,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:52,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:52,503 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:52,503 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:52,503 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:52,503 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:52,503 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:52,504 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,504 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:52,504 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:52,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:52,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:52,504 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:52,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:52,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:52,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:52,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:52,505 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:52,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:52,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,507 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,507 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:52,507 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:52,507 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:52,507 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:52,507 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:52,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:52,507 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,507 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,507 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:52,507 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:52,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:52,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:52,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:52,507 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:52,507 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:52,508 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:52,608 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:52,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:52,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:52,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:52,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:52,611 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:52,611 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:52,611 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:52,611 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:52,612 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,612 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:52,612 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:52,612 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:52,613 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:52,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:52,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32786 deadline: 1681995292612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,613 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,613 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:52,613 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:52,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:52,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32782 deadline: 1681995292613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:52,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:52,614 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:52,614 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:52,614 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:52,614 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:52,614 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:52,614 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:52,614 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:52,614 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:52,614 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:52,614 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:52,614 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:52,615 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:52,615 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:52,615 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:52,615 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,615 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:52,615 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:52,616 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,616 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:52,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:52,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:52,616 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,616 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:52,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:52,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:52,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:52,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:52,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7999073b[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:52,617 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:52,617 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,618 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,618 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:52,618 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:52,618 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:52,618 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,618 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:52,619 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:52,619 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32792 deadline: 1681995292616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,619 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:52,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:52,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:52,619 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:52,619 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:52,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:52,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:52,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:52,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:52,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:52,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,621 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:52,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60994 deadline: 1681995292617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:52,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:52,621 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:52,621 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:52,621 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:52,621 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:52,621 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:52,622 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,622 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:52,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,622 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:52,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:52,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:52,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:52,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:52,623 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:52,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:52,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:52,624 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:52,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:52,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,626 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,626 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:52,626 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:52,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:52,626 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:52,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:52,626 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:52,626 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:52,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:52,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:52,626 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:52,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:52,626 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:52,626 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:52,626 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,627 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:52,627 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,627 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,714 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=795 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/e128a06d96634e8bb6e57329d757fe0e 2023-04-20 12:53:52,721 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:52,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32832 deadline: 1681995292719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,727 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:52,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:52,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:52,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:52,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:52,729 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:52,730 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:52,730 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:52,730 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:52,730 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,730 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:52,731 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:52,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:52,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:52,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:52,732 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:52,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:52,732 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:52,732 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:52,732 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:52,732 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,732 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:52,732 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:52,732 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:52,732 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:52,732 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:52,732 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:52,732 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:52,732 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,733 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:52,733 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:52,733 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:52,733 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:52,733 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:52,733 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,733 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:52,733 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:52,734 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,734 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:52,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:52,734 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:52,734 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:52,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:52,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:52,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:52,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:52,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74972792[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:52,735 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:52,735 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,736 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,736 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:52,736 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:52,736 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:52,736 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,736 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:52,737 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,737 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:52,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:52,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:52,737 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:52,737 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:52,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:52,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:52,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:52,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:52,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:52,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:52,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:52,738 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:52,739 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:52,739 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:52,739 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:52,739 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:52,740 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:52,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:52,740 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,740 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:52,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:52,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,740 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:52,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:52,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:52,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:52,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:52,742 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:52,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:52,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,743 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:52,743 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:52,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:52,743 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,743 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:52,743 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:52,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:52,743 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,743 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,743 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,743 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,743 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:52,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:52,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:52,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:52,744 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:52,744 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:52,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:52,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:52,844 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:52,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:52,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:52,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:52,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:52,846 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:52,846 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:52,846 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:52,846 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:52,847 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,847 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:52,847 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,847 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:52,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:52,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:52,848 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,848 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:52,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:52,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:52,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:52,848 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:52,849 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:52,849 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:52,849 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:52,849 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:52,849 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:52,849 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:52,849 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:52,849 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:52,849 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:52,849 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:52,849 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:52,849 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:52,849 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:52,849 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,849 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:52,849 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:52,851 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,851 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:52,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:52,851 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,851 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:52,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:52,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:52,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:52,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:52,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:52,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18c52010[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:52,852 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:52,852 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,853 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,853 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:52,853 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:52,853 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:52,853 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,853 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:52,854 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:52,854 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:52,854 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:52,854 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:52,854 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:52,854 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:52,854 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:52,854 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:52,854 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:52,855 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,855 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,855 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:52,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:52,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:52,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:52,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:52,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:52,855 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:52,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:52,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:52,856 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:52,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:52,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,858 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,858 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:52,858 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:52,858 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,858 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:52,858 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:52,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:52,858 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,858 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:52,858 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:52,858 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,858 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:52,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:52,858 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:52,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:52,858 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:52,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:52,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:52,914 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:52,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32782 deadline: 1681995292914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,919 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:52,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32786 deadline: 1681995292916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,922 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:52,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32792 deadline: 1681995292922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,926 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:52,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60994 deadline: 1681995292923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,958 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:52,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:52,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:52,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:52,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:52,961 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:52,961 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:52,961 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:52,961 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:52,962 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,962 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:52,962 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,962 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:52,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:52,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:52,962 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,963 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:52,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:52,963 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:52,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:52,963 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:52,963 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:52,963 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:52,964 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:52,964 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:52,964 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:52,964 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:52,964 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:52,964 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:52,964 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:52,964 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:52,964 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:52,964 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:52,964 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:52,964 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,964 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:52,964 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:52,967 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,967 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:52,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:52,967 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,967 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:52,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:52,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:52,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:52,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:52,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:52,968 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:52,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31cecf7d[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:52,968 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,969 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,969 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:52,969 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,969 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:52,969 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:52,969 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:52,970 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,970 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:52,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:52,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:52,970 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:52,973 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:52,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:52,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:52,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:52,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:52,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:52,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:52,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:52,978 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:52,978 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:52,978 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:52,978 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:52,978 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:52,979 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,979 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,979 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:52,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:52,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:52,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:52,980 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:52,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:52,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:52,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:52,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:52,981 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:52,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:52,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,982 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,982 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:52,982 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:52,982 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:52,982 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,982 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:52,982 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:52,982 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:52,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:52,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:52,982 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:52,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:52,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:52,982 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:52,983 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:52,983 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:52,983 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:52,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:52,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:53,083 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:53,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:53,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:53,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:53,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:53,086 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:53,086 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:53,086 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:53,086 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:53,086 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:53,086 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:53,087 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,087 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:53,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:53,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:53,087 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,087 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:53,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:53,087 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:53,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:53,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:53,088 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:53,088 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:53,088 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:53,088 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:53,088 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:53,088 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:53,088 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:53,088 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:53,088 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:53,088 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:53,088 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:53,088 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:53,089 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:53,089 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:53,089 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,089 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:53,089 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:53,089 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,089 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:53,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:53,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:53,090 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:53,090 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:53,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:53,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:53,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:53,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:53,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:53,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dfa6a0[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:53,091 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:53,091 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:53,092 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:53,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:53,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:53,092 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:53,092 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:53,092 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:53,092 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,092 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:53,093 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:53,093 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,093 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:53,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:53,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:53,093 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:53,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:53,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:53,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:53,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:53,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:53,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:53,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:53,095 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:53,095 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:53,095 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:53,095 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:53,095 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:53,096 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:53,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:53,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:53,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:53,096 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:53,097 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:53,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:53,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:53,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:53,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:53,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:53,097 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:53,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:53,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:53,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:53,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,098 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:53,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:53,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,100 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,100 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:53,100 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:53,100 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:53,100 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,100 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:53,100 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:53,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:53,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:53,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:53,100 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:53,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:53,100 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:53,100 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:53,100 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:53,100 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:53,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:53,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:53,126 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=795 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/dad322557f66472ea42e0df4a925f516 2023-04-20 12:53:53,200 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:53,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:53,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:53,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:53,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:53,203 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:53,203 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:53,203 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:53,203 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:53,204 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:53,204 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:53,204 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,204 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:53,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:53,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:53,205 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,205 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:53,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:53,205 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:53,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:53,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:53,206 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:53,206 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:53,206 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:53,206 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:53,206 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:53,206 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:53,206 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:53,206 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:53,206 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:53,206 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:53,206 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:53,206 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:53,206 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:53,207 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:53,207 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,207 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:53,207 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:53,207 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:53,208 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:53,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:53,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:53,208 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:53,208 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:53,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:53,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:53,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:53,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:53,209 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:53,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d39ba9e[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:53,209 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:53,210 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:53,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:53,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:53,210 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:53,210 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:53,210 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:53,210 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,210 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:53,211 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:53,211 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:53,211 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,211 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:53,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:53,213 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:53,213 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:53,213 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:53,213 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:53,213 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:53,214 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:53,214 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:53,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:53,214 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:53,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:53,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:53,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:53,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:53,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:53,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:53,214 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:53,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:53,215 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:53,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:53,216 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,216 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:53,216 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:53,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:53,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:53,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:53,217 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:53,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:53,217 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:53,217 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:53,217 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,217 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:53,217 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:53,217 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:53,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:53,217 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:53,217 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:53,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:53,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:53,317 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:53,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:53,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:53,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:53,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:53,320 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:53,320 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:53,320 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:53,320 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:53,324 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:53,324 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:53,325 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,325 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:53,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:53,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:53,325 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,325 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:53,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:53,325 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:53,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:53,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:53,326 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:53,326 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:53,326 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:53,326 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:53,326 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:53,326 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:53,326 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:53,326 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:53,326 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:53,326 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:53,327 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:53,327 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:53,327 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:53,327 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:53,327 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,327 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:53,327 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:53,328 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,328 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:53,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:53,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:53,328 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:53,328 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:53,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:53,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:53,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:53,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:53,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:53,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ccdd689[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:53,329 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:53,329 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:53,330 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:53,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:53,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:53,330 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:53,330 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:53,330 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:53,330 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,330 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:53,330 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,331 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:53,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:53,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:53,331 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:53,331 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:53,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:53,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:53,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:53,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:53,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:53,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:53,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:53,333 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:53,333 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:53,333 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:53,333 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:53,333 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:53,334 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:53,334 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:53,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:53,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:53,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:53,334 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:53,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:53,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:53,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:53,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:53,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:53,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:53,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:53,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:53,335 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:53,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:53,336 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:53,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,338 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,338 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:53,338 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:53,338 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:53,338 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:53,338 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,338 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:53,338 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:53,338 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:53,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:53,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:53,338 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:53,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:53,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:53,338 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:53,338 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:53,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:53,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:53,422 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:53,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32782 deadline: 1681995293417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,423 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:53,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32786 deadline: 1681995293420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,428 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:53,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32792 deadline: 1681995293427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,434 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:53,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60994 deadline: 1681995293429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,438 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:53,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:53,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:53,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:53,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:53,446 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:53,446 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:53,446 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:53,446 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:53,447 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:53,447 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:53,448 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:53,448 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:53,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:53,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:53,448 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,448 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:53,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:53,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:53,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:53,449 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:53,449 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:53,449 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:53,449 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:53,449 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:53,449 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:53,454 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:53,454 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:53,454 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:53,454 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:53,454 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:53,454 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:53,454 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:53,454 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:53,454 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,454 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:53,454 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:53,463 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,463 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:53,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:53,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:53,463 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:53,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:53,463 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:53,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:53,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:53,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:53,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:53,465 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:53,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bbf4b4a[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:53,465 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:53,468 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:53,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:53,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:53,468 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:53,468 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:53,468 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:53,468 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,468 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:53,469 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:53,469 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:53,469 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,469 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:53,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:53,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:53,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:53,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:53,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:53,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:53,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:53,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:53,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:53,471 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:53,471 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:53,471 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:53,471 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:53,471 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:53,472 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:53,472 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:53,472 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:53,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:53,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:53,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:53,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:53,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:53,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:53,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:53,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:53,472 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:53,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:53,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:53,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:53,473 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:53,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:53,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,475 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:53,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:53,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:53,475 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:53,475 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:53,475 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:53,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:53,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:53,475 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:53,475 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,476 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:53,476 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:53,476 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:53,476 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:53,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:53,476 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:53,476 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:53,476 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:53,539 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=795 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/5671d40814264aebbe71aa0da47beb00 2023-04-20 12:53:53,546 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/e128a06d96634e8bb6e57329d757fe0e as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/e128a06d96634e8bb6e57329d757fe0e 2023-04-20 12:53:53,552 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/e128a06d96634e8bb6e57329d757fe0e, entries=250, sequenceid=795, filesize=16.7 K 2023-04-20 12:53:53,553 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/dad322557f66472ea42e0df4a925f516 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/dad322557f66472ea42e0df4a925f516 2023-04-20 12:53:53,562 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/dad322557f66472ea42e0df4a925f516, entries=150, sequenceid=795, filesize=11.9 K 2023-04-20 12:53:53,563 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/5671d40814264aebbe71aa0da47beb00 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/5671d40814264aebbe71aa0da47beb00 2023-04-20 12:53:53,571 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/5671d40814264aebbe71aa0da47beb00, entries=150, sequenceid=795, filesize=11.9 K 2023-04-20 12:53:53,572 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~445.88 KB/456576, currentSize=26.84 KB/27480 for fc3812e05598851193449f5684adf036 in 1267ms, sequenceid=795, compaction requested=true 2023-04-20 12:53:53,572 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:53,572 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:53:53,572 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-20 12:53:53,573 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-20 12:53:53,572 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:53:53,573 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:53:53,574 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 66560 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-20 12:53:53,575 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): fc3812e05598851193449f5684adf036/A is initiating minor compaction (all files) 2023-04-20 12:53:53,575 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fc3812e05598851193449f5684adf036/A in TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:53,575 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/6759b824d89340a09fd4a2208ddcedb9, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/7f9286b9a9db424ebda6519b8c85ba84, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/01426621474b40eab00989d3de456745, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/3a6988e4560d47e6829169d3dedd7569, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/e128a06d96634e8bb6e57329d757fe0e] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp, totalSize=65 K 2023-04-20 12:53:53,575 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 6759b824d89340a09fd4a2208ddcedb9, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=699, earliestPutTs=1681995228394 2023-04-20 12:53:53,575 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61680 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-20 12:53:53,575 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): fc3812e05598851193449f5684adf036/B is initiating minor compaction (all files) 2023-04-20 12:53:53,576 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 7f9286b9a9db424ebda6519b8c85ba84, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=714, earliestPutTs=1681995228452 2023-04-20 12:53:53,576 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fc3812e05598851193449f5684adf036/B in TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:53,576 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/a214f6ecfda44a83a3bffc62c428aca7, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/23d0442755e640599fd3ff90ce861c42, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/5bb9db7c1cd842ef87f1cfed6e7ef1f0, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/f59fcb4473e642269925310c4701c745, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/dad322557f66472ea42e0df4a925f516] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp, totalSize=60.2 K 2023-04-20 12:53:53,576 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:53,576 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 01426621474b40eab00989d3de456745, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=749, earliestPutTs=1681995229545 2023-04-20 12:53:53,576 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting a214f6ecfda44a83a3bffc62c428aca7, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=699, earliestPutTs=1681995228394 2023-04-20 12:53:53,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:53,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:53,579 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 3a6988e4560d47e6829169d3dedd7569, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=758, earliestPutTs=1681995230589 2023-04-20 12:53:53,579 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 23d0442755e640599fd3ff90ce861c42, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=714, earliestPutTs=1681995228452 2023-04-20 12:53:53,579 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting e128a06d96634e8bb6e57329d757fe0e, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=795, earliestPutTs=1681995231695 2023-04-20 12:53:53,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:53,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:53,580 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:53,580 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:53,580 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:53,580 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:53,580 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:53,580 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 5bb9db7c1cd842ef87f1cfed6e7ef1f0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=749, earliestPutTs=1681995229545 2023-04-20 12:53:53,580 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:53,581 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting f59fcb4473e642269925310c4701c745, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=758, earliestPutTs=1681995230589 2023-04-20 12:53:53,581 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting dad322557f66472ea42e0df4a925f516, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=795, earliestPutTs=1681995231695 2023-04-20 12:53:53,581 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:53,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:53,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:53,581 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:53,582 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:53,582 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:53,584 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:53,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:53,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:53,584 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:53,585 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:53,585 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:53,585 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:53,585 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:53,585 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:53,586 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:53,586 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:53,586 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:53,586 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:53,586 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fc3812e05598851193449f5684adf036 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-20 12:53:53,586 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=A 2023-04-20 12:53:53,586 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:53:53,587 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=B 2023-04-20 12:53:53,587 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:53:53,587 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=C 2023-04-20 12:53:53,587 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:53:53,596 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): fc3812e05598851193449f5684adf036#A#compaction#79 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:53:53,604 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): fc3812e05598851193449f5684adf036#B#compaction#81 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:53:53,646 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/ff28ec31a4e24e23b15756a037f0d896 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/ff28ec31a4e24e23b15756a037f0d896 2023-04-20 12:53:53,653 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/d2b7d80bafa0416dbf090160c5411bfe as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/d2b7d80bafa0416dbf090160c5411bfe 2023-04-20 12:53:53,659 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in fc3812e05598851193449f5684adf036/A of fc3812e05598851193449f5684adf036 into ff28ec31a4e24e23b15756a037f0d896(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:53:53,659 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:53,660 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036., storeName=fc3812e05598851193449f5684adf036/A, priority=11, startTime=1681995233572; duration=0sec 2023-04-20 12:53:53,660 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:53:53,660 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-20 12:53:53,662 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73764 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-20 12:53:53,662 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): fc3812e05598851193449f5684adf036/C is initiating minor compaction (all files) 2023-04-20 12:53:53,662 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fc3812e05598851193449f5684adf036/C in TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:53,663 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/ced07e9101ae4c429c0b773207715999, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/2fc2928bc712431c8de9a78335e480b6, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/69e6ebe7b57e4062b788b93869f71b2f, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/ac914b06475449d49bacd87ca89646ea, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/2118ba1f99a249a58d3f13969814c1cd, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/5671d40814264aebbe71aa0da47beb00] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp, totalSize=72.0 K 2023-04-20 12:53:53,663 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in fc3812e05598851193449f5684adf036/B of fc3812e05598851193449f5684adf036 into d2b7d80bafa0416dbf090160c5411bfe(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:53:53,663 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:53,663 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036., storeName=fc3812e05598851193449f5684adf036/B, priority=11, startTime=1681995233572; duration=0sec 2023-04-20 12:53:53,663 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:53:53,664 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting ced07e9101ae4c429c0b773207715999, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=664, earliestPutTs=1681995227804 2023-04-20 12:53:53,664 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 2fc2928bc712431c8de9a78335e480b6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=699, earliestPutTs=1681995228394 2023-04-20 12:53:53,665 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 69e6ebe7b57e4062b788b93869f71b2f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=714, earliestPutTs=1681995228452 2023-04-20 12:53:53,665 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting ac914b06475449d49bacd87ca89646ea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=749, earliestPutTs=1681995229545 2023-04-20 12:53:53,665 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 2118ba1f99a249a58d3f13969814c1cd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=758, earliestPutTs=1681995230589 2023-04-20 12:53:53,666 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 5671d40814264aebbe71aa0da47beb00, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=795, earliestPutTs=1681995231695 2023-04-20 12:53:53,676 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): fc3812e05598851193449f5684adf036#C#compaction#82 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:53:54,028 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=802 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/8d378d1628394ae78de516138f4050f4 2023-04-20 12:53:54,087 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/0e9f011979044e8fbbbb1d94703a3890 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/0e9f011979044e8fbbbb1d94703a3890 2023-04-20 12:53:54,092 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in fc3812e05598851193449f5684adf036/C of fc3812e05598851193449f5684adf036 into 0e9f011979044e8fbbbb1d94703a3890(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:53:54,092 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:54,092 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036., storeName=fc3812e05598851193449f5684adf036/C, priority=10, startTime=1681995233573; duration=0sec 2023-04-20 12:53:54,092 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:53:54,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on fc3812e05598851193449f5684adf036 2023-04-20 12:53:54,429 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:54,439 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=802 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/0a10eb7e252541eba72194f18802b833 2023-04-20 12:53:54,442 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1681995234431, max timestamp=1681995234441Num uniques 859; 2023-04-20 12:53:54,442 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-20 12:53:54,442 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1681995234431, max timestamp=1681995234441Num uniques 859; 2023-04-20 12:53:54,442 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-04-20 12:53:54,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681995234433, max timestamp=1681995234441Num uniques 859; 2023-04-20 12:53:54,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-20 12:53:54,451 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=802 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/667275278d6c4cf89f592bb40992dbe9 2023-04-20 12:53:54,456 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:54,456 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/8d378d1628394ae78de516138f4050f4 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/8d378d1628394ae78de516138f4050f4 2023-04-20 12:53:54,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32786 deadline: 1681995294454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,456 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:54,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32782 deadline: 1681995294454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,459 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:54,459 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:54,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32792 deadline: 1681995294454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60994 deadline: 1681995294455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,461 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/8d378d1628394ae78de516138f4050f4, entries=100, sequenceid=802, filesize=9.5 K 2023-04-20 12:53:54,462 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/0a10eb7e252541eba72194f18802b833 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/0a10eb7e252541eba72194f18802b833 2023-04-20 12:53:54,467 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/0a10eb7e252541eba72194f18802b833, entries=100, sequenceid=802, filesize=9.5 K 2023-04-20 12:53:54,468 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/667275278d6c4cf89f592bb40992dbe9 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/667275278d6c4cf89f592bb40992dbe9 2023-04-20 12:53:54,474 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/667275278d6c4cf89f592bb40992dbe9, entries=100, sequenceid=802, filesize=9.5 K 2023-04-20 12:53:54,475 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=161.02 KB/164880 for fc3812e05598851193449f5684adf036 in 889ms, sequenceid=802, compaction requested=false 2023-04-20 12:53:54,475 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:54,476 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:54,476 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:54,476 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:54,476 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,481 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:54,481 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:54,483 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,483 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:54,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:54,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:54,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:54,484 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:54,484 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:54,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:54,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:54,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:54,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:54,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46a8103b[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:54,485 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:54,485 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:54,486 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:54,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:54,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:54,486 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:54,486 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:54,486 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:54,487 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,487 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:54,489 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,489 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:54,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,489 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:54,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:54,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:54,489 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:54,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:54,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:54,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:54,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:54,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:54,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:54,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:54,492 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:54,492 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:54,492 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:54,492 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:54,492 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:54,493 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:54,493 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:54,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:54,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:54,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:54,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:54,493 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:54,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:54,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:54,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:54,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:54,494 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:54,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:54,494 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:54,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:54,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:54,496 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,496 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:54,496 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:54,496 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:54,496 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:54,496 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:54,496 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:54,496 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:54,496 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:54,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:54,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:54,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:54,496 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:54,496 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:54,497 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:54,497 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:54,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:54,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:54,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on fc3812e05598851193449f5684adf036 2023-04-20 12:53:54,559 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fc3812e05598851193449f5684adf036 3/3 column families, dataSize=167.72 KB heapSize=466.55 KB 2023-04-20 12:53:54,560 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681995234453, max timestamp=1681995234558Num uniques 1268; 2023-04-20 12:53:54,560 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=A 2023-04-20 12:53:54,560 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=58068, new segment data size=7688, suffix heap size=159840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1268, new segment cells count=168 2023-04-20 12:53:54,560 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:53:54,560 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=B 2023-04-20 12:53:54,560 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681995234453, max timestamp=1681995234558Num uniques 1118; 2023-04-20 12:53:54,561 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-20 12:53:54,561 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=7688, suffix heap size=144240,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=168 2023-04-20 12:53:54,561 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=C 2023-04-20 12:53:54,561 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:53:54,577 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:54,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32786 deadline: 1681995294574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,577 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:54,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60994 deadline: 1681995294575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,577 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:54,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32792 deadline: 1681995294576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,578 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:54,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32782 deadline: 1681995294577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,597 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:54,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:54,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:54,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:54,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:54,600 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:54,600 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:54,600 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:54,600 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:54,600 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:54,600 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:54,601 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:54,601 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:54,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:54,601 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,601 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:54,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:54,601 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:54,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:54,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:54,602 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:54,602 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:54,602 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:54,602 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:54,602 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:54,602 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:54,602 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:54,602 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:54,602 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:54,602 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:54,602 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:54,602 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:54,602 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:54,602 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:54,603 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,603 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:54,603 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:54,603 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,603 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:54,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:54,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:54,604 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:54,604 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:54,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:54,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:54,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:54,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:54,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:54,605 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:54,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46c5bf3a[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:54,605 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:54,605 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:54,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:54,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:54,605 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:54,606 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:54,606 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,606 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:54,606 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:54,606 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,606 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:54,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:54,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:54,606 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:54,606 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:54,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:54,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:54,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:54,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:54,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:54,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:54,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:54,608 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:54,609 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:54,609 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:54,609 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:54,609 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:54,609 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:54,610 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:54,609 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:54,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:54,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:54,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:54,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:54,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:54,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:54,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:54,610 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:54,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:54,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:54,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:54,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:54,611 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:54,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:54,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,612 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,613 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:54,613 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:54,613 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:54,613 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:54,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:54,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:54,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:54,613 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:54,613 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:54,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:54,613 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:54,613 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:54,613 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:54,613 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:54,613 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:54,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:54,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:54,679 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:54,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32786 deadline: 1681995294678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,679 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:54,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60994 deadline: 1681995294678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,682 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:54,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32792 deadline: 1681995294678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,682 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:54,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32782 deadline: 1681995294679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,713 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:54,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:54,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:54,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:54,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:54,716 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:54,716 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:54,716 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:54,716 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:54,717 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:54,717 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:54,717 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:54,717 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:54,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:54,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:54,718 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,718 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:54,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:54,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:54,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:54,719 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:54,719 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:54,719 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:54,719 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:54,719 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:54,719 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:54,720 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:54,720 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:54,720 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:54,720 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:54,720 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:54,720 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:54,720 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:54,720 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:54,720 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,720 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:54,720 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:54,721 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,721 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:54,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:54,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:54,722 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:54,722 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:54,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:54,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:54,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:54,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:54,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:54,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54ae40f5[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:54,723 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:54,723 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:54,724 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:54,724 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:54,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:54,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32832 deadline: 1681995294722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:54,724 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:54,724 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:54,724 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:54,725 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,725 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:54,726 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4127 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036., hostname=jenkins-hbase20.apache.org,36277,1681995213947, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-20 12:53:54,726 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:54,726 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:54,727 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:54,727 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:54,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:54,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:54,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:54,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:54,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:54,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:54,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:54,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:54,730 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:54,730 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:54,730 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:54,730 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:54,730 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:54,734 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:54,734 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:54,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:54,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:54,734 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:54,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:54,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:54,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:54,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:54,734 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:54,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:54,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:54,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:54,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:54,735 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:54,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:54,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:54,738 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:54,738 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:54,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:54,738 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:54,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:54,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:54,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:54,738 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:54,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:54,738 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,738 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:54,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:54,738 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:54,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:54,738 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:54,739 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,739 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:54,739 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:54,739 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:54,839 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:54,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:54,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:54,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:54,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:54,841 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:54,841 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:54,841 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:54,841 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:54,842 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:54,842 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:54,842 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,842 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:54,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:54,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:54,842 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,843 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:54,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:54,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:54,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:54,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:54,843 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:54,843 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:54,843 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:54,844 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:54,844 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:54,844 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:54,844 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:54,844 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:54,844 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:54,844 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:54,844 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:54,844 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:54,844 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:54,844 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:54,844 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,844 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:54,844 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:54,845 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,845 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:54,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:54,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:54,845 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:54,845 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:54,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:54,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:54,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:54,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:54,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:54,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@150aa431[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:54,846 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:54,846 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:54,847 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:54,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:54,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:54,847 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:54,847 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:54,847 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:54,847 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,847 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:54,848 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,848 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:54,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:54,848 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:54,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:54,848 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:54,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:54,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:54,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:54,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:54,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:54,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:54,850 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:54,850 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:54,850 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:54,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:54,850 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:54,850 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:54,851 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:54,851 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:54,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:54,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:54,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:54,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:54,851 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:54,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:54,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:54,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:54,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:54,852 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:54,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:54,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:54,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:54,853 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:54,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:54,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,855 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:54,855 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:54,855 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:54,855 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:54,855 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,855 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:54,855 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:54,855 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,855 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:54,855 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:54,855 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:54,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:54,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:54,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:54,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:54,856 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:54,856 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:54,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:54,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:54,881 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:54,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60994 deadline: 1681995294880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,882 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:54,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32786 deadline: 1681995294881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,884 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:54,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32792 deadline: 1681995294884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,885 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:54,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32782 deadline: 1681995294884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,956 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:54,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:54,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:54,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:54,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:54,959 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:54,959 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:54,959 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:54,959 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:54,959 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:54,959 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:54,960 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,960 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:54,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:54,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:54,960 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,960 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:54,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:54,961 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:54,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:54,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:54,961 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:54,961 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:54,961 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:54,961 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:54,961 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:54,961 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:54,961 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:54,962 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:54,962 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:54,962 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:54,962 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:54,962 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:54,962 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:54,962 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:54,962 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,962 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:54,962 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:54,963 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,963 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:54,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:54,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:54,963 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:54,963 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:54,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:54,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:54,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:54,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:54,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:54,964 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:54,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13255763[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:54,964 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:54,965 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:54,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:54,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:54,965 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,965 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:54,965 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:54,965 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:54,965 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:54,966 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:54,966 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:54,966 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:54,967 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:54,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:54,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:54,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:54,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:54,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:54,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:54,968 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=844 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/ad240b4ba20a4298bf062b4de2148a45 2023-04-20 12:53:54,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:54,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:54,969 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:54,969 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:54,969 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:54,969 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:54,969 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:54,970 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:54,970 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:54,971 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:54,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:54,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:54,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:54,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:54,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:54,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:54,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:54,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:54,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:54,971 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:54,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:54,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:54,973 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:54,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:54,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,975 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:54,975 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,975 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:54,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:54,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:54,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:54,975 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:54,975 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:54,975 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:54,975 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:54,975 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:54,976 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:54,976 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:54,976 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:54,976 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:54,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:54,976 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:54,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:54,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:55,076 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:55,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:55,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:55,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:55,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:55,079 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:55,079 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:55,079 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:55,079 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:55,079 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,079 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:55,080 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:55,080 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:55,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:55,080 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,080 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:55,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:55,081 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:55,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:55,081 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:55,081 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:55,081 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:55,081 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:55,081 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:55,081 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:55,081 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:55,081 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:55,081 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:55,081 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:55,081 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:55,082 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:55,082 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:55,082 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:55,082 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,082 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:55,082 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:55,082 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,082 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:55,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:55,083 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,083 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:55,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:55,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:55,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:55,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:55,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:55,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78f21652[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:55,084 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:55,084 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,085 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,085 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:55,085 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:55,085 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:55,085 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,085 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:55,086 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,086 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:55,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:55,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:55,086 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:55,086 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:55,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:55,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:55,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:55,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:55,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:55,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:55,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:55,088 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:55,088 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:55,088 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:55,088 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:55,088 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:55,089 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,089 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:55,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:55,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:55,090 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,090 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,092 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:55,092 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:55,092 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:55,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:55,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:55,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:55,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:55,092 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:55,092 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:55,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:55,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:55,093 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,093 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:55,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:55,093 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:55,093 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:55,093 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,093 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,093 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:55,093 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:55,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:55,094 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:55,184 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:55,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32786 deadline: 1681995295184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,185 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:55,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60994 deadline: 1681995295184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,188 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:55,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32792 deadline: 1681995295185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,192 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:55,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:55,194 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:55,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32782 deadline: 1681995295192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:55,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:55,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:55,199 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:55,199 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:55,199 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:55,199 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:55,200 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,200 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:55,200 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:55,200 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:55,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:55,201 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,201 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:55,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:55,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:55,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:55,201 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:55,201 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:55,201 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:55,201 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:55,201 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:55,201 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:55,202 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:55,202 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:55,202 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:55,202 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:55,202 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:55,202 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:55,202 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:55,202 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:55,202 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,202 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:55,202 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:55,203 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,203 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:55,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:55,203 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,203 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:55,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:55,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:55,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:55,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:55,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:55,204 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:55,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5efd5851[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:55,204 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,205 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,205 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:55,205 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:55,205 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:55,205 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,205 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:55,206 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:55,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:55,206 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:55,206 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:55,206 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:55,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:55,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:55,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:55,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:55,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:55,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:55,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:55,208 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:55,208 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:55,208 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:55,208 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:55,208 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:55,210 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,210 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,210 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:55,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:55,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:55,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:55,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:55,210 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:55,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:55,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:55,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:55,212 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:55,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:55,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,214 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,214 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:55,214 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:55,214 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:55,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:55,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:55,214 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:55,214 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,214 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,214 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,214 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:55,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:55,214 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:55,214 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:55,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:55,214 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:55,214 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:55,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:55,315 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:55,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:55,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:55,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:55,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:55,318 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:55,318 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:55,318 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:55,318 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:55,318 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,318 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:55,319 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,319 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:55,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:55,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:55,319 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,319 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:55,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:55,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:55,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:55,320 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:55,320 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:55,320 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:55,320 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:55,320 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:55,320 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:55,320 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:55,320 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:55,320 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:55,320 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:55,320 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:55,320 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:55,320 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:55,321 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:55,321 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,321 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:55,321 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:55,321 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,321 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:55,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:55,322 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,322 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:55,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:55,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:55,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:55,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:55,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:55,323 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:55,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e164493[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:55,323 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,323 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,323 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:55,324 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:55,324 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:55,324 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,324 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:55,324 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,324 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:55,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:55,324 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:55,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:55,325 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:55,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:55,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:55,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:55,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:55,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:55,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:55,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:55,326 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:55,326 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:55,326 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:55,326 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:55,326 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:55,327 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,327 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:55,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,327 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:55,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:55,328 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:55,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:55,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:55,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:55,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:55,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:55,332 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:55,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:55,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,334 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:55,334 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,334 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:55,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:55,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:55,334 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:55,334 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:55,334 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,334 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,334 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,334 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,334 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:55,334 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:55,334 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:55,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:55,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:55,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:55,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:55,335 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:55,388 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60.38 KB at sequenceid=844 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/3f9d1036026744e983641ed2753ba955 2023-04-20 12:53:55,434 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:55,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:55,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:55,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:55,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:55,437 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:55,437 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:55,437 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:55,437 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:55,438 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,438 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:55,439 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,439 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:55,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:55,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:55,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:55,439 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,439 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:55,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:55,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:55,440 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:55,450 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:55,450 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:55,450 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:55,450 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:55,450 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:55,457 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:55,457 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:55,457 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:55,457 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:55,457 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:55,457 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:55,458 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:55,458 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:55,458 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,458 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:55,458 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:55,459 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,459 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:55,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:55,459 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,459 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:55,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:55,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:55,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:55,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:55,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:55,460 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:55,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2731967d[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:55,460 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,461 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,461 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:55,461 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,461 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:55,461 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:55,461 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:55,462 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:55,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:55,462 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:55,462 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:55,462 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:55,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:55,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:55,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:55,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:55,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:55,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:55,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:55,464 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:55,464 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:55,464 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:55,464 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:55,464 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:55,465 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,465 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:55,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:55,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,465 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:55,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:55,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:55,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:55,466 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:55,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:55,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:55,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,468 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:55,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:55,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,469 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,469 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:55,469 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:55,469 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:55,469 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,469 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:55,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:55,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:55,470 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:55,470 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,470 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,470 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:55,470 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:55,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:55,470 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:55,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:55,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:55,570 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:55,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:55,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:55,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:55,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:55,573 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:55,573 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:55,573 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:55,573 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:55,574 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,574 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:55,574 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:55,574 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:55,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:55,575 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,575 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:55,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:55,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:55,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:55,575 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:55,575 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:55,576 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:55,576 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:55,576 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:55,576 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:55,576 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:55,576 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:55,576 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:55,576 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:55,576 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:55,576 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:55,576 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:55,576 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:55,576 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,576 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:55,576 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:55,577 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:55,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:55,577 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:55,577 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,577 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:55,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:55,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:55,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:55,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:55,578 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:55,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f99444f[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:55,578 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,579 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,579 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:55,579 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:55,579 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,579 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:55,579 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:55,580 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,580 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:55,580 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:55,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:55,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:55,580 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:55,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:55,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:55,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:55,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:55,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:55,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:55,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:55,582 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:55,582 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:55,582 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:55,582 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:55,582 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:55,583 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,583 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:55,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:55,583 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:55,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:55,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:55,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:55,584 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:55,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:55,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:55,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,585 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:55,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:55,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,586 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,586 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:55,586 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:55,586 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,586 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,586 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,586 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:55,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:55,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:55,586 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:55,586 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,586 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:55,587 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:55,587 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:55,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:55,587 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:55,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:55,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:55,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:55,687 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:55,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:55,688 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:55,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32786 deadline: 1681995295687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:55,690 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:55,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60994 deadline: 1681995295690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:55,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:55,691 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:55,691 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:55,691 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:55,691 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:55,691 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,691 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:55,692 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:55,692 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:55,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:55,693 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,693 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:55,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:55,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:55,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:55,694 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:55,694 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:55,694 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:55,694 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:55,694 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:55,694 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:55,694 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:55,694 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:55,694 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:55,694 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:55,694 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:55,694 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:55,695 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:55,695 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:55,695 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:55,695 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32792 deadline: 1681995295690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,695 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:55,695 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:55,695 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:55,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32782 deadline: 1681995295695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,696 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,696 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:55,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:55,696 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,696 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:55,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:55,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:55,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:55,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:55,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:55,698 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:55,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b92a23d[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:55,698 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,698 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,698 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:55,698 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:55,698 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:55,698 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,698 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:55,699 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:55,699 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:55,699 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,699 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:55,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:55,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:55,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:55,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:55,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:55,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:55,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:55,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:55,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:55,701 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:55,701 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:55,701 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:55,701 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:55,701 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:55,702 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,702 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:55,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:55,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,702 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:55,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:55,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:55,703 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:55,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:55,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:55,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:55,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,704 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:55,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:55,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,705 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,705 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:55,705 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:55,705 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,705 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:55,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:55,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:55,705 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,705 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:55,705 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:55,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:55,706 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:55,705 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:55,706 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:55,706 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:55,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:55,805 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=844 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/3426a67328f94f3aa6bd60deba73f639 2023-04-20 12:53:55,806 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:55,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:55,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:55,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:55,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:55,809 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:55,809 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:55,809 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:55,809 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:55,810 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,810 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:55,811 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/ad240b4ba20a4298bf062b4de2148a45 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/ad240b4ba20a4298bf062b4de2148a45 2023-04-20 12:53:55,811 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,811 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:55,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:55,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:55,812 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,812 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:55,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:55,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:55,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:55,812 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:55,813 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:55,813 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:55,813 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:55,813 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:55,813 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:55,813 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:55,813 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:55,813 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:55,813 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:55,813 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:55,813 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:55,814 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:55,814 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:55,814 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,814 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:55,814 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:55,815 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,815 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:55,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:55,815 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,815 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:55,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:55,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:55,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:55,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:55,818 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/ad240b4ba20a4298bf062b4de2148a45, entries=200, sequenceid=844, filesize=14.3 K 2023-04-20 12:53:55,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:55,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@570c4300[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:55,818 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:55,818 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,818 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/3f9d1036026744e983641ed2753ba955 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/3f9d1036026744e983641ed2753ba955 2023-04-20 12:53:55,819 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,819 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:55,819 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:55,819 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:55,819 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,819 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:55,820 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,820 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:55,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:55,820 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:55,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:55,820 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:55,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:55,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:55,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:55,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:55,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:55,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:55,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:55,823 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:55,823 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:55,823 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:55,824 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:55,824 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:55,825 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/3f9d1036026744e983641ed2753ba955, entries=150, sequenceid=844, filesize=11.9 K 2023-04-20 12:53:55,833 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,833 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,833 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:55,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:55,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:55,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:55,833 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/3426a67328f94f3aa6bd60deba73f639 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/3426a67328f94f3aa6bd60deba73f639 2023-04-20 12:53:55,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:55,834 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:55,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:55,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:55,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:55,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,835 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:55,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:55,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,838 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,838 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:55,838 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:55,838 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:55,838 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:55,838 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:55,838 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,838 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,838 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:55,838 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:55,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:55,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:55,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:55,838 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:55,839 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:55,839 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:55,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:55,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:55,840 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/3426a67328f94f3aa6bd60deba73f639, entries=150, sequenceid=844, filesize=11.9 K 2023-04-20 12:53:55,840 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~82.74 KB/84730, heapSize ~224.06 KB/229440, currentSize=100.63 KB/103050 for fc3812e05598851193449f5684adf036 in 1281ms, sequenceid=844, compaction requested=true 2023-04-20 12:53:55,841 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:55,841 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:53:55,841 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:53:55,841 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:53:55,841 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:53:55,841 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:53:55,842 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35034 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:53:55,842 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37474 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:53:55,842 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): fc3812e05598851193449f5684adf036/B is initiating minor compaction (all files) 2023-04-20 12:53:55,842 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): fc3812e05598851193449f5684adf036/A is initiating minor compaction (all files) 2023-04-20 12:53:55,842 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fc3812e05598851193449f5684adf036/B in TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:55,842 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fc3812e05598851193449f5684adf036/A in TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:55,842 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/ff28ec31a4e24e23b15756a037f0d896, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/8d378d1628394ae78de516138f4050f4, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/ad240b4ba20a4298bf062b4de2148a45] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp, totalSize=36.6 K 2023-04-20 12:53:55,842 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/d2b7d80bafa0416dbf090160c5411bfe, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/0a10eb7e252541eba72194f18802b833, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/3f9d1036026744e983641ed2753ba955] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp, totalSize=34.2 K 2023-04-20 12:53:55,843 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting ff28ec31a4e24e23b15756a037f0d896, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=795, earliestPutTs=1681995231695 2023-04-20 12:53:55,843 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting d2b7d80bafa0416dbf090160c5411bfe, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=795, earliestPutTs=1681995231695 2023-04-20 12:53:55,843 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 8d378d1628394ae78de516138f4050f4, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=802, earliestPutTs=1681995232309 2023-04-20 12:53:55,843 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 0a10eb7e252541eba72194f18802b833, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=802, earliestPutTs=1681995232309 2023-04-20 12:53:55,843 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting ad240b4ba20a4298bf062b4de2148a45, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=844, earliestPutTs=1681995234453 2023-04-20 12:53:55,844 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 3f9d1036026744e983641ed2753ba955, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=844, earliestPutTs=1681995234453 2023-04-20 12:53:55,851 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): fc3812e05598851193449f5684adf036#A#compaction#88 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:53:55,851 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): fc3812e05598851193449f5684adf036#B#compaction#89 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:53:55,939 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:55,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:55,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:55,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:55,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:55,944 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:55,944 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:55,944 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:55,944 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:55,945 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,945 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:55,945 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,945 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:55,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:55,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:55,946 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:55,946 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:55,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:55,946 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:55,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:55,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:55,946 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:55,947 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:55,947 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:55,947 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:55,947 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:55,947 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:55,948 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:55,948 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:55,948 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:55,948 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:55,949 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fc3812e05598851193449f5684adf036 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-20 12:53:55,949 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=A 2023-04-20 12:53:55,949 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:53:55,949 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=B 2023-04-20 12:53:55,949 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:53:55,949 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=C 2023-04-20 12:53:55,949 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:53:56,265 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/6e6beeb295d84d94a3c6ebc09e6e12f4 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/6e6beeb295d84d94a3c6ebc09e6e12f4 2023-04-20 12:53:56,265 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/71aca374bec44b81989b14843390a16b as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/71aca374bec44b81989b14843390a16b 2023-04-20 12:53:56,270 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fc3812e05598851193449f5684adf036/B of fc3812e05598851193449f5684adf036 into 71aca374bec44b81989b14843390a16b(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:53:56,270 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:56,270 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fc3812e05598851193449f5684adf036/A of fc3812e05598851193449f5684adf036 into 6e6beeb295d84d94a3c6ebc09e6e12f4(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:53:56,270 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036., storeName=fc3812e05598851193449f5684adf036/B, priority=13, startTime=1681995235841; duration=0sec 2023-04-20 12:53:56,270 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:56,270 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036., storeName=fc3812e05598851193449f5684adf036/A, priority=13, startTime=1681995235841; duration=0sec 2023-04-20 12:53:56,270 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:53:56,271 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:53:56,271 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:53:56,272 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34966 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:53:56,272 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): fc3812e05598851193449f5684adf036/C is initiating minor compaction (all files) 2023-04-20 12:53:56,272 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fc3812e05598851193449f5684adf036/C in TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:56,272 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/0e9f011979044e8fbbbb1d94703a3890, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/667275278d6c4cf89f592bb40992dbe9, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/3426a67328f94f3aa6bd60deba73f639] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp, totalSize=34.1 K 2023-04-20 12:53:56,272 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 0e9f011979044e8fbbbb1d94703a3890, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=795, earliestPutTs=1681995231695 2023-04-20 12:53:56,273 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 667275278d6c4cf89f592bb40992dbe9, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=802, earliestPutTs=1681995232309 2023-04-20 12:53:56,273 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 3426a67328f94f3aa6bd60deba73f639, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=844, earliestPutTs=1681995234453 2023-04-20 12:53:56,280 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): fc3812e05598851193449f5684adf036#C#compaction#91 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:53:56,307 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/f366ed4667864bab8f6942927760a4f9 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/f366ed4667864bab8f6942927760a4f9 2023-04-20 12:53:56,312 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fc3812e05598851193449f5684adf036/C of fc3812e05598851193449f5684adf036 into f366ed4667864bab8f6942927760a4f9(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:53:56,313 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:56,313 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036., storeName=fc3812e05598851193449f5684adf036/C, priority=13, startTime=1681995235841; duration=0sec 2023-04-20 12:53:56,313 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:53:56,356 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=862 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/cb84af0e3ea34068aedde842aaecbec2 2023-04-20 12:53:56,367 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=862 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/4b45f240e3c242f58ba1378e6ceac8c3 2023-04-20 12:53:56,385 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=862 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/30d4f66f1c9f4fb6a8667aa094238960 2023-04-20 12:53:56,392 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/cb84af0e3ea34068aedde842aaecbec2 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/cb84af0e3ea34068aedde842aaecbec2 2023-04-20 12:53:56,398 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/cb84af0e3ea34068aedde842aaecbec2, entries=150, sequenceid=862, filesize=11.9 K 2023-04-20 12:53:56,400 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/4b45f240e3c242f58ba1378e6ceac8c3 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/4b45f240e3c242f58ba1378e6ceac8c3 2023-04-20 12:53:56,405 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/4b45f240e3c242f58ba1378e6ceac8c3, entries=150, sequenceid=862, filesize=11.9 K 2023-04-20 12:53:56,406 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/30d4f66f1c9f4fb6a8667aa094238960 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/30d4f66f1c9f4fb6a8667aa094238960 2023-04-20 12:53:56,412 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/30d4f66f1c9f4fb6a8667aa094238960, entries=150, sequenceid=862, filesize=11.9 K 2023-04-20 12:53:56,413 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=0 B/0 for fc3812e05598851193449f5684adf036 in 465ms, sequenceid=862, compaction requested=false 2023-04-20 12:53:56,413 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:56,413 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:56,413 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:56,413 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:56,413 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,414 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:56,414 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:56,415 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,415 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:56,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:56,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:56,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:56,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:56,416 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:56,416 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:56,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:56,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:56,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:56,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ca5a5e2[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:56,418 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:56,418 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:56,419 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:56,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:56,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:56,419 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:56,419 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,419 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:56,419 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:56,419 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:56,421 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:56,421 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:56,421 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:56,421 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:56,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:56,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:56,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:56,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:56,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:56,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:56,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:56,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:56,424 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:56,424 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:56,424 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:56,424 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:56,424 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:56,425 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:56,425 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:56,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:56,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:56,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:56,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:56,425 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:56,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:56,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:56,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:56,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:56,426 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:56,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:56,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:56,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:56,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,428 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:56,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:56,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,438 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,438 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:56,438 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:56,438 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:56,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:56,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:56,438 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,438 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:56,438 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:56,438 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:56,438 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:56,438 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:56,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:56,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:56,438 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:56,438 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:56,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:56,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:56,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:56,539 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:56,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:56,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:56,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:56,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:56,546 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:56,546 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:56,546 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:56,546 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:56,546 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:56,547 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:56,547 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,547 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:56,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:56,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:56,548 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,548 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:56,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:56,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:56,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:56,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:56,548 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:56,549 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:56,549 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:56,549 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:56,549 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:56,549 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:56,549 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:56,549 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:56,549 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:56,549 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:56,549 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:56,549 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:56,549 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:56,549 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:56,549 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,550 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:56,550 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:56,551 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,551 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:56,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:56,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:56,552 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:56,552 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:56,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:56,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:56,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:56,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:56,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:56,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@799b3d05[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:56,553 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:56,553 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:56,554 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:56,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:56,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:56,554 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:56,554 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:56,554 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,554 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:56,554 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:56,555 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,555 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:56,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:56,555 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:56,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:56,555 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:56,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:56,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:56,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:56,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:56,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:56,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:56,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:56,558 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:56,558 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:56,558 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:56,558 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:56,558 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:56,559 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:56,559 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:56,559 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:56,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:56,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:56,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:56,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:56,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:56,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:56,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:56,559 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:56,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:56,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:56,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:56,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:56,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,561 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:56,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:56,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,562 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,562 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:56,562 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:56,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:56,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:56,562 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,562 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:56,562 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:56,562 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:56,563 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:56,562 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:56,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:56,562 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:56,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:56,563 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:56,563 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:56,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:56,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:56,663 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:56,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:56,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:56,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:56,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:56,665 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:56,665 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:56,665 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:56,665 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:56,666 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:56,666 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:56,666 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:56,666 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:56,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:56,667 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,667 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:56,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:56,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:56,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:56,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:56,668 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:56,668 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:56,668 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:56,668 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:56,668 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:56,668 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:56,668 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:56,668 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:56,668 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:56,668 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:56,668 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:56,669 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:56,669 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:56,669 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:56,669 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,669 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:56,669 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:56,670 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,670 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:56,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:56,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:56,670 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:56,670 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:56,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:56,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:56,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:56,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:56,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:56,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17f6fb41[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:56,671 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:56,671 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:56,672 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:56,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:56,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:56,672 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:56,672 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:56,672 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:56,672 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,672 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:56,673 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:56,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:56,673 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:56,673 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:56,673 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:56,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:56,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:56,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:56,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:56,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:56,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:56,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:56,675 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:56,675 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:56,675 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:56,676 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:56,676 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:56,676 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:56,676 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:56,677 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:56,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:56,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:56,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:56,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:56,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:56,677 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:56,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:56,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:56,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:56,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:56,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:56,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:56,678 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:56,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:56,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,680 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,680 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:56,680 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:56,680 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:56,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:56,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:56,680 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:56,680 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:56,680 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,680 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:56,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:56,680 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:56,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:56,680 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:56,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:56,680 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:56,680 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:56,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:56,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:56,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on fc3812e05598851193449f5684adf036 2023-04-20 12:53:56,702 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing fc3812e05598851193449f5684adf036 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-20 12:53:56,704 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=A 2023-04-20 12:53:56,704 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:53:56,704 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=B 2023-04-20 12:53:56,704 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:53:56,704 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=C 2023-04-20 12:53:56,704 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:53:56,719 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:56,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32792 deadline: 1681995296717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,720 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:56,720 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681995236715, max timestamp=1681995236719Num uniques 859; 2023-04-20 12:53:56,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32786 deadline: 1681995296719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,720 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-20 12:53:56,720 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681995236715, max timestamp=1681995236719Num uniques 859; 2023-04-20 12:53:56,720 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-20 12:53:56,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681995236715, max timestamp=1681995236719Num uniques 859; 2023-04-20 12:53:56,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-20 12:53:56,738 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:56,738 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:56,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32782 deadline: 1681995296735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60994 deadline: 1681995296736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,780 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:56,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:56,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:56,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:56,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:56,783 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:56,783 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:56,783 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:56,783 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:56,784 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:56,784 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:56,784 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,784 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:56,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:56,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:56,785 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,785 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:56,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:56,785 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:56,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:56,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:56,785 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:56,786 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:56,786 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:56,786 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:56,786 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:56,786 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:56,786 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:56,786 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:56,786 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:56,786 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:56,786 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:56,786 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:56,786 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:56,786 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,786 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:56,786 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:56,787 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,787 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:56,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:56,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:56,787 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:56,787 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:56,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:56,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:56,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:56,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:56,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:56,789 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:56,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2257befc[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:56,789 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:56,789 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:56,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:56,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:56,789 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,789 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:56,789 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:56,789 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:56,789 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:56,790 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,790 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:56,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:56,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:56,790 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:56,790 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:56,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:56,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:56,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:56,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:56,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:56,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:56,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:56,792 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:56,792 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:56,792 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:56,792 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:56,792 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:56,793 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:56,793 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:56,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:56,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:56,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:56,793 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:56,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:56,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:56,793 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:56,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:56,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:56,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:56,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:56,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:56,794 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:56,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:56,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:56,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,796 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,796 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:56,796 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:56,796 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:56,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:56,796 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:56,796 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:56,796 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:56,796 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:56,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:56,796 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:56,796 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:56,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:56,796 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:56,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:56,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:56,798 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:56,798 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:56,822 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:56,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32792 deadline: 1681995296821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,823 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:56,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32786 deadline: 1681995296821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,840 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:56,840 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:56,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60994 deadline: 1681995296839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32782 deadline: 1681995296839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,898 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:56,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:56,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:56,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:56,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:56,901 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:56,901 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:56,901 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:56,901 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:56,901 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:56,901 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:56,902 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,902 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:56,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:56,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:56,902 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,902 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:56,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:56,903 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:56,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:56,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:56,903 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:56,903 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:56,903 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:56,903 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:56,903 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:56,903 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:56,903 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:56,903 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:56,903 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:56,903 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:56,904 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:56,904 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:56,904 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:56,904 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:56,904 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,904 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:56,904 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:56,904 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:56,905 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:56,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:56,905 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:56,905 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:56,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:56,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:56,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:56,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:56,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:56,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@730c85e1[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:56,906 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:56,906 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:56,907 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:56,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:56,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:56,907 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,907 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:56,907 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:56,907 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:56,907 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:56,908 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,908 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:56,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:56,909 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:56,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:56,909 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:56,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:56,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:56,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:56,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:56,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:56,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:56,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:56,911 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:56,911 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:56,911 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:56,911 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:56,911 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:56,912 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:56,912 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:56,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:56,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:56,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:56,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:56,912 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:56,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:56,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:56,912 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:56,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:56,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:56,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:56,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:56,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:56,913 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:56,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:56,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,915 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,915 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:56,915 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:56,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:56,915 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:56,915 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:56,915 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:56,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:56,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:56,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:56,915 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:56,915 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:56,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:56,915 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:56,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:56,915 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:56,915 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:56,918 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:56,918 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:57,018 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:57,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:57,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:57,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:57,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:57,021 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:57,021 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:57,021 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:57,021 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:57,021 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,021 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:57,022 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,022 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:57,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:57,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:57,022 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:57,022 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:57,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:57,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:57,023 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:57,023 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:57,023 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:57,024 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:57,024 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:57,024 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:57,024 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:57,024 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:57,024 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:57,024 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:57,024 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:57,024 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:57,024 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:57,024 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:57,024 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,024 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:57,024 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:57,025 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,025 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:57,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:57,025 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,026 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:57,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:57,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:57,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:57,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:57,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:57,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53dffd90[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:57,027 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:57,027 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,027 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,028 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:57,028 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:57,028 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:57,028 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,028 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:57,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32786 deadline: 1681995297024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,028 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:57,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32792 deadline: 1681995297024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,029 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:57,029 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:57,029 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:57,029 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:57,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:57,028 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:57,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:57,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:57,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:57,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:57,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:57,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:57,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:57,033 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:57,033 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:57,033 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:57,033 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:57,033 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:57,034 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,034 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:57,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:57,034 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:57,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:57,035 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:57,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:57,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:57,036 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:57,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:57,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:57,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:57,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,038 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,038 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:57,038 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:57,038 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:57,038 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:57,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:57,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:57,038 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:57,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:57,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:57,038 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:57,038 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:57,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:57,038 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:57,038 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,038 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:57,038 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,042 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:57,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60994 deadline: 1681995297041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,044 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:57,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32782 deadline: 1681995297042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,112 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=879 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/836e0a83f4984fc49c00f471c84b17aa 2023-04-20 12:53:57,139 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:57,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:57,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:57,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:57,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:57,142 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:57,142 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:57,142 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:57,142 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:57,142 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,142 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:57,143 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,143 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:57,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:57,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:57,143 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,143 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:57,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:57,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:57,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:57,144 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:57,144 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:57,144 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:57,144 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:57,144 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:57,145 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:57,145 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:57,145 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:57,145 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:57,145 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:57,145 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:57,145 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:57,145 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:57,145 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:57,145 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,145 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:57,145 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:57,146 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,146 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:57,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:57,146 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,146 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:57,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:57,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:57,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:57,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:57,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:57,147 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:57,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73d32859[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:57,147 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,148 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,148 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:57,148 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:57,148 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:57,148 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,148 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:57,149 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,149 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:57,149 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:57,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,149 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:57,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:57,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:57,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:57,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:57,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:57,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:57,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:57,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:57,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:57,151 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:57,151 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:57,151 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:57,151 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:57,151 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:57,152 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,152 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:57,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:57,152 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:57,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:57,154 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:57,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:57,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:57,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:57,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:57,155 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:57,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:57,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,156 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:57,156 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:57,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:57,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:57,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:57,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:57,157 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,157 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:57,157 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:57,157 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,157 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,157 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,157 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,157 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:57,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:57,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:57,158 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:57,158 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:57,158 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:57,258 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:57,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:57,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:57,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:57,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:57,261 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:57,261 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:57,261 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:57,261 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:57,261 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,262 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:57,262 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:57,262 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:57,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:57,262 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,263 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:57,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:57,263 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:57,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:57,263 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:57,263 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:57,263 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:57,263 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:57,263 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:57,263 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:57,263 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:57,264 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:57,264 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:57,264 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:57,264 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:57,264 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:57,264 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:57,264 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:57,264 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,264 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:57,264 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:57,265 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,265 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:57,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:57,265 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,265 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:57,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:57,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:57,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:57,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:57,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:57,266 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:57,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cecd881[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:57,266 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,267 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,267 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:57,267 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:57,267 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:57,267 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,267 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:57,267 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,268 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:57,268 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:57,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:57,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:57,268 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:57,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:57,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:57,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:57,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:57,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:57,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:57,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:57,270 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:57,270 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:57,270 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:57,270 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:57,270 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:57,271 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,271 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,271 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:57,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:57,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:57,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:57,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:57,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:57,272 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:57,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:57,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:57,273 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:57,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:57,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,275 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:57,275 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:57,275 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:57,275 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:57,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:57,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:57,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:57,275 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:57,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:57,275 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:57,275 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:57,275 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:57,275 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,275 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:57,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:57,275 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,275 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,329 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:57,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32786 deadline: 1681995297329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,330 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:57,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32792 deadline: 1681995297329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,344 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:57,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 314 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60994 deadline: 1681995297343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,348 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:57,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32782 deadline: 1681995297345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,375 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:57,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:57,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:57,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:57,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:57,378 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:57,378 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:57,378 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:57,378 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:57,378 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,378 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:57,379 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:57,379 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:57,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:57,379 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,379 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:57,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:57,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:57,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:57,380 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:57,380 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:57,380 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:57,380 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:57,380 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:57,380 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:57,380 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:57,381 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:57,381 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:57,381 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:57,381 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:57,381 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:57,381 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:57,381 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:57,381 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,381 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:57,381 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:57,382 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,382 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:57,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:57,382 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,382 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:57,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:57,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:57,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:57,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:57,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:57,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b586b70[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:57,383 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:57,383 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,384 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,384 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:57,384 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:57,384 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:57,384 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,384 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:57,385 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:57,385 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:57,385 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,385 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:57,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:57,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:57,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:57,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:57,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:57,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:57,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:57,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:57,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:57,387 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:57,387 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:57,387 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:57,387 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:57,387 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:57,388 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,388 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,388 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:57,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:57,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:57,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,388 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:57,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:57,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:57,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:57,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:57,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:57,389 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:57,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:57,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,391 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,391 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:57,391 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:57,391 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:57,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:57,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:57,391 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:57,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:57,391 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:57,391 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:57,391 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:57,391 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:57,391 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,391 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:57,391 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:57,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:57,491 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:57,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:57,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:57,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:57,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:57,494 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:57,494 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:57,494 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:57,494 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:57,495 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,495 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:57,495 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,495 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:57,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:57,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:57,496 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,496 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:57,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:57,496 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:57,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:57,496 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:57,496 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:57,496 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:57,496 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:57,496 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:57,496 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:57,497 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:57,497 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:57,497 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:57,497 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:57,497 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:57,497 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:57,497 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:57,497 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:57,497 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,497 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:57,497 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:57,498 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,498 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:57,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:57,498 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,498 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:57,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:57,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:57,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:57,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:57,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:57,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57c72af6[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:57,499 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:57,499 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,500 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,500 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:57,500 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:57,500 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:57,500 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,500 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:57,501 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,501 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:57,501 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:57,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:57,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:57,501 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:57,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:57,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:57,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:57,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:57,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:57,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:57,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:57,503 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:57,503 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:57,503 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:57,503 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:57,503 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:57,503 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,503 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,504 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:57,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:57,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:57,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:57,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:57,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:57,504 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:57,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:57,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:57,505 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:57,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:57,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,506 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,506 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:57,506 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:57,506 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:57,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:57,506 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:57,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:57,506 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:57,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:57,506 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,507 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:57,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:57,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:57,507 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,507 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,507 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:57,507 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:57,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:57,524 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=879 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/ebb5e025f03b44a9b9cf95138aceff8b 2023-04-20 12:53:57,607 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:57,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:57,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:57,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:57,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:57,609 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:57,609 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:57,609 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:57,609 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:57,610 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,610 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:57,610 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:57,610 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:57,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:57,611 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,611 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:57,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:57,611 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:57,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:57,612 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:57,612 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:57,612 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:57,612 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:57,612 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:57,612 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:57,612 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:57,612 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:57,612 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:57,612 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:57,612 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:57,612 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:57,612 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:57,613 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:57,613 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,613 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:57,613 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:57,614 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,614 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:57,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:57,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:57,615 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,615 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:57,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:57,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:57,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:57,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:57,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3dbc41b7[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:57,616 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:57,616 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,617 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,617 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,617 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:57,618 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:57,618 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:57,618 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:57,619 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,619 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:57,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:57,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:57,619 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:57,619 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:57,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:57,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:57,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:57,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:57,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:57,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:57,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:57,621 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:57,621 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:57,621 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:57,621 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:57,621 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:57,622 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,622 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,622 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:57,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:57,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:57,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:57,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:57,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:57,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:57,629 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:57,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:57,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:57,630 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:57,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,638 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,638 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:57,638 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:57,638 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:57,638 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:57,638 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:57,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:57,638 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:57,638 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:57,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:57,638 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,638 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:57,638 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,638 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:57,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:57,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:57,739 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:57,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:57,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:57,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:57,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:57,741 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:57,741 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:57,741 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:57,741 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:57,742 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,742 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:57,743 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:57,743 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:57,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:57,743 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,743 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:57,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:57,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:57,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:57,743 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:57,744 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:57,744 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:57,744 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:57,744 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:57,744 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:57,744 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:57,744 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:57,744 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:57,744 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:57,744 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:57,744 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:57,744 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:57,744 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:57,744 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,744 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:57,745 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:57,745 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,745 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:57,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:57,746 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,746 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:57,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:57,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:57,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:57,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:57,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:57,747 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:57,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@371ebe51[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:57,747 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,747 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,747 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:57,747 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:57,747 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,747 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:57,747 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:57,748 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,748 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:57,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:57,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:57,748 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:57,748 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:57,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:57,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:57,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:57,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:57,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:57,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:57,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:57,750 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:57,750 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:57,750 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:57,750 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:57,750 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:57,751 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,751 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,751 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:57,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:57,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:57,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,751 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:57,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:57,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:57,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:57,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:57,752 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:57,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:57,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:57,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,753 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,753 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:57,754 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:57,754 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,754 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,754 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,754 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:57,754 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:57,754 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:57,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:57,754 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:57,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:57,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:57,754 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:57,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:57,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:57,754 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:57,754 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:57,832 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:57,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32792 deadline: 1681995297831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,836 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:57,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32786 deadline: 1681995297833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,849 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:57,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 316 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60994 deadline: 1681995297845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,852 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:53:57,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:32782 deadline: 1681995297851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,854 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:57,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:57,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:57,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:57,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:57,857 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:57,857 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:57,857 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:57,857 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:57,857 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,858 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:57,858 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,858 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:57,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:57,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:57,858 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,858 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:57,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:57,859 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:57,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:57,859 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:57,859 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:57,859 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:57,859 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:57,859 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:57,859 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:57,860 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:57,860 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:57,860 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:57,860 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:57,860 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:57,860 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:57,860 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:57,860 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:57,860 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,860 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:57,860 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:57,861 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,861 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:57,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:57,861 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,861 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:57,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:57,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:57,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:57,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:57,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:57,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fa54029[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:57,862 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:57,862 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,863 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,863 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:57,863 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:57,863 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,863 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:57,863 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:57,864 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,864 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:57,864 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:57,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:57,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:57,864 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:57,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:57,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:57,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:57,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:57,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:57,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:57,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:57,865 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:57,865 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:57,865 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:57,865 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:57,866 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:57,866 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,866 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:57,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:57,866 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:57,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:57,867 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:57,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:57,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:57,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:57,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:57,868 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:57,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:57,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,869 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,870 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:57,870 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:57,870 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,870 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:57,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:57,870 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:57,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:57,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:57,870 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:57,870 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:57,870 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:57,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:57,870 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:57,870 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:57,870 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:57,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:57,935 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=879 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/fa3b99aa76cf4696a9eb4871db878a0d 2023-04-20 12:53:57,941 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/836e0a83f4984fc49c00f471c84b17aa as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/836e0a83f4984fc49c00f471c84b17aa 2023-04-20 12:53:57,946 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/836e0a83f4984fc49c00f471c84b17aa, entries=250, sequenceid=879, filesize=16.7 K 2023-04-20 12:53:57,947 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/ebb5e025f03b44a9b9cf95138aceff8b as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/ebb5e025f03b44a9b9cf95138aceff8b 2023-04-20 12:53:57,953 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/ebb5e025f03b44a9b9cf95138aceff8b, entries=150, sequenceid=879, filesize=11.9 K 2023-04-20 12:53:57,954 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/fa3b99aa76cf4696a9eb4871db878a0d as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/fa3b99aa76cf4696a9eb4871db878a0d 2023-04-20 12:53:57,959 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/fa3b99aa76cf4696a9eb4871db878a0d, entries=150, sequenceid=879, filesize=11.9 K 2023-04-20 12:53:57,959 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for fc3812e05598851193449f5684adf036 in 1257ms, sequenceid=879, compaction requested=true 2023-04-20 12:53:57,959 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:57,960 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:53:57,960 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:53:57,960 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:53:57,960 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:53:57,960 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:53:57,960 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37580 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:53:57,961 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): fc3812e05598851193449f5684adf036/B is initiating minor compaction (all files) 2023-04-20 12:53:57,961 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42460 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:53:57,961 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fc3812e05598851193449f5684adf036/B in TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:57,961 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): fc3812e05598851193449f5684adf036/A is initiating minor compaction (all files) 2023-04-20 12:53:57,961 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/71aca374bec44b81989b14843390a16b, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/4b45f240e3c242f58ba1378e6ceac8c3, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/ebb5e025f03b44a9b9cf95138aceff8b] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp, totalSize=36.7 K 2023-04-20 12:53:57,961 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of fc3812e05598851193449f5684adf036/A in TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:57,961 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/6e6beeb295d84d94a3c6ebc09e6e12f4, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/cb84af0e3ea34068aedde842aaecbec2, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/836e0a83f4984fc49c00f471c84b17aa] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp, totalSize=41.5 K 2023-04-20 12:53:57,961 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 71aca374bec44b81989b14843390a16b, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=844, earliestPutTs=1681995234453 2023-04-20 12:53:57,961 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 6e6beeb295d84d94a3c6ebc09e6e12f4, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=844, earliestPutTs=1681995234453 2023-04-20 12:53:57,961 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 4b45f240e3c242f58ba1378e6ceac8c3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=862, earliestPutTs=1681995234569 2023-04-20 12:53:57,962 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting cb84af0e3ea34068aedde842aaecbec2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=862, earliestPutTs=1681995234569 2023-04-20 12:53:57,962 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting ebb5e025f03b44a9b9cf95138aceff8b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=879, earliestPutTs=1681995236699 2023-04-20 12:53:57,962 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 836e0a83f4984fc49c00f471c84b17aa, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=879, earliestPutTs=1681995236699 2023-04-20 12:53:57,968 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): fc3812e05598851193449f5684adf036#B#compaction#97 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:53:57,969 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): fc3812e05598851193449f5684adf036#A#compaction#98 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:53:57,970 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:57,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:57,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:53:57,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:53:57,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:53:57,980 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:53:57,980 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:57,980 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:53:57,980 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:53:57,981 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,981 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:57,981 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:57,981 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:57,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:57,981 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:57,981 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:53:57,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:57,982 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:57,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:53:57,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:57,982 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:53:57,982 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:53:57,982 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:53:57,983 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:53:57,983 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:53:57,983 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:53:57,986 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:53:57,986 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:53:57,986 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:57,986 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. started... 2023-04-20 12:53:57,986 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing fc3812e05598851193449f5684adf036 3/3 column families, dataSize=120.76 KB heapSize=335.77 KB 2023-04-20 12:53:57,986 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=A 2023-04-20 12:53:57,986 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:53:57,986 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=B 2023-04-20 12:53:57,986 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:53:57,986 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=C 2023-04-20 12:53:57,986 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:53:58,064 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7abbc496 to 127.0.0.1:56673 2023-04-20 12:53:58,064 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x11a028ee to 127.0.0.1:56673 2023-04-20 12:53:58,064 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:53:58,064 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:53:58,068 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x21551c39 to 127.0.0.1:56673 2023-04-20 12:53:58,068 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:53:58,069 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x373daaaa to 127.0.0.1:56673 2023-04-20 12:53:58,069 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:53:58,390 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/5534fdf5991b45b599c287a9384ebc08 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/5534fdf5991b45b599c287a9384ebc08 2023-04-20 12:53:58,392 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/ea28721b65ac4205adb4ba5e5f566f8b as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/ea28721b65ac4205adb4ba5e5f566f8b 2023-04-20 12:53:58,394 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=912 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/656af3ca25f04368847b3f151a85fe03 2023-04-20 12:53:58,397 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fc3812e05598851193449f5684adf036/B of fc3812e05598851193449f5684adf036 into ea28721b65ac4205adb4ba5e5f566f8b(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:53:58,397 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fc3812e05598851193449f5684adf036/A of fc3812e05598851193449f5684adf036 into 5534fdf5991b45b599c287a9384ebc08(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:53:58,397 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:58,397 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:58,398 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036., storeName=fc3812e05598851193449f5684adf036/B, priority=13, startTime=1681995237960; duration=0sec 2023-04-20 12:53:58,398 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036., storeName=fc3812e05598851193449f5684adf036/A, priority=13, startTime=1681995237960; duration=0sec 2023-04-20 12:53:58,398 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:53:58,398 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:53:58,398 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:53:58,399 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37512 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:53:58,399 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): fc3812e05598851193449f5684adf036/C is initiating minor compaction (all files) 2023-04-20 12:53:58,399 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of fc3812e05598851193449f5684adf036/C in TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:58,399 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/f366ed4667864bab8f6942927760a4f9, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/30d4f66f1c9f4fb6a8667aa094238960, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/fa3b99aa76cf4696a9eb4871db878a0d] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp, totalSize=36.6 K 2023-04-20 12:53:58,400 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting f366ed4667864bab8f6942927760a4f9, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=844, earliestPutTs=1681995234453 2023-04-20 12:53:58,400 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 30d4f66f1c9f4fb6a8667aa094238960, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=862, earliestPutTs=1681995234569 2023-04-20 12:53:58,400 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting fa3b99aa76cf4696a9eb4871db878a0d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=879, earliestPutTs=1681995236699 2023-04-20 12:53:58,407 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): fc3812e05598851193449f5684adf036#C#compaction#101 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:53:58,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on fc3812e05598851193449f5684adf036 2023-04-20 12:53:58,768 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. as already flushing 2023-04-20 12:53:58,768 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5d114eb9 to 127.0.0.1:56673 2023-04-20 12:53:58,769 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:53:58,807 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=912 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/d9c663aa7fc34ec4b682c1378cb95025 2023-04-20 12:53:58,821 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/780153d01a5c4f4fa3126bbe3efcbf12 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/780153d01a5c4f4fa3126bbe3efcbf12 2023-04-20 12:53:58,827 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in fc3812e05598851193449f5684adf036/C of fc3812e05598851193449f5684adf036 into 780153d01a5c4f4fa3126bbe3efcbf12(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:53:58,827 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:58,828 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036., storeName=fc3812e05598851193449f5684adf036/C, priority=13, startTime=1681995237960; duration=0sec 2023-04-20 12:53:58,828 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:53:58,836 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3627406c to 127.0.0.1:56673 2023-04-20 12:53:58,836 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:53:58,842 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x542f272a to 127.0.0.1:56673 2023-04-20 12:53:58,842 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:53:58,856 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x04d5a66f to 127.0.0.1:56673 2023-04-20 12:53:58,857 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:53:58,857 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7d6b0f87 to 127.0.0.1:56673 2023-04-20 12:53:58,858 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:53:59,227 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=912 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/369ddf384c484c0b89c4293fb7b22001 2023-04-20 12:53:59,235 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/656af3ca25f04368847b3f151a85fe03 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/656af3ca25f04368847b3f151a85fe03 2023-04-20 12:53:59,243 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/656af3ca25f04368847b3f151a85fe03, entries=150, sequenceid=912, filesize=11.9 K 2023-04-20 12:53:59,244 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/d9c663aa7fc34ec4b682c1378cb95025 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/d9c663aa7fc34ec4b682c1378cb95025 2023-04-20 12:53:59,249 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/d9c663aa7fc34ec4b682c1378cb95025, entries=150, sequenceid=912, filesize=11.9 K 2023-04-20 12:53:59,250 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/369ddf384c484c0b89c4293fb7b22001 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/369ddf384c484c0b89c4293fb7b22001 2023-04-20 12:53:59,255 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/369ddf384c484c0b89c4293fb7b22001, entries=150, sequenceid=912, filesize=11.9 K 2023-04-20 12:53:59,255 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~335.72 KB/343776, currentSize=33.54 KB/34350 for fc3812e05598851193449f5684adf036 in 1269ms, sequenceid=912, compaction requested=false 2023-04-20 12:53:59,255 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:53:59,255 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:59,256 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:53:59,256 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:53:59,256 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:59,256 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:53:59,256 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:53:59,257 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:59,257 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:59,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:59,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:59,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:59,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:59,257 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:59,257 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:53:59,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:59,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:59,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:59,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:59,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:53:59,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61db4bb1[Count = 0] remaining members to acquire global barrier 2023-04-20 12:53:59,258 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:53:59,258 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:59,259 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:59,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:59,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:59,259 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:53:59,259 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:53:59,259 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:59,259 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:53:59,259 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:53:59,260 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:53:59,260 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:59,260 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:59,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:59,260 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:53:59,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:59,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:59,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:59,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:59,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:59,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:59,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:59,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:59,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:59,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:53:59,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:53:59,262 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:53:59,262 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:53:59,262 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:53:59,262 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:53:59,262 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:53:59,263 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:59,263 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:59,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:59,263 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:59,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:59,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:59,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:53:59,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:53:59,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:59,264 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:59,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:59,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:53:59,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:59,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:59,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:53:59,264 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:59,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:59,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:59,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:53:59,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:53:59,266 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:53:59,266 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:59,266 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:53:59,266 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:59,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:53:59,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:53:59,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:53:59,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:53:59,267 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:53:59,266 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:53:59,267 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:53:59,266 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:53:59,267 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:53:59,267 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:59,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:53:59,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:53:59,267 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:59,267 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:53:59,267 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:53:59,367 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:53:59,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:53:59,368 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-20 12:53:59,369 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(394): wrote 152 2023-04-20 12:53:59,369 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(394): wrote 123 2023-04-20 12:53:59,369 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(394): wrote 187 2023-04-20 12:53:59,369 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(394): wrote 200 2023-04-20 12:53:59,369 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(394): wrote 151 2023-04-20 12:53:59,369 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-20 12:53:59,369 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(398): read 5947 2023-04-20 12:53:59,369 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(398): read 5412 2023-04-20 12:53:59,369 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-20 12:53:59,369 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(402): scanned 2400 2023-04-20 12:53:59,369 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(403): verified 7198 rows 2023-04-20 12:53:59,369 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(402): scanned 2394 2023-04-20 12:53:59,369 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(403): verified 7180 rows 2023-04-20 12:53:59,370 INFO [Listener at localhost.localdomain/42205] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-20 12:53:59,370 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x21c17a8e to 127.0.0.1:56673 2023-04-20 12:53:59,370 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:53:59,376 INFO [Listener at localhost.localdomain/42205] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-20 12:53:59,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-20 12:53:59,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-20 12:53:59,389 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681995239389"}]},"ts":"1681995239389"} 2023-04-20 12:53:59,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-20 12:53:59,390 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-20 12:53:59,392 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-20 12:53:59,394 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=fc3812e05598851193449f5684adf036, UNASSIGN}] 2023-04-20 12:53:59,395 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=fc3812e05598851193449f5684adf036, UNASSIGN 2023-04-20 12:53:59,396 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=fc3812e05598851193449f5684adf036, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:53:59,396 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681995239396"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681995239396"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681995239396"}]},"ts":"1681995239396"} 2023-04-20 12:53:59,397 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947}] 2023-04-20 12:53:59,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-20 12:53:59,558 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close fc3812e05598851193449f5684adf036 2023-04-20 12:53:59,558 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing fc3812e05598851193449f5684adf036, disabling compactions & flushes 2023-04-20 12:53:59,558 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:59,558 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:59,558 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. after waiting 0 ms 2023-04-20 12:53:59,558 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:53:59,558 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing fc3812e05598851193449f5684adf036 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-20 12:53:59,559 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=A 2023-04-20 12:53:59,559 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:53:59,559 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=B 2023-04-20 12:53:59,560 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:53:59,560 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK fc3812e05598851193449f5684adf036, store=C 2023-04-20 12:53:59,560 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:53:59,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-20 12:53:59,975 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=923 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/9e24d40cf4cf454fa5cf0877c91f86ed 2023-04-20 12:53:59,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-20 12:54:00,395 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=923 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/215e7857cc9842f6ab66f8c82cec3980 2023-04-20 12:54:00,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-20 12:54:00,815 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=923 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/06d6a20e2c224230b701587e49d78cbf 2023-04-20 12:54:00,825 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/A/9e24d40cf4cf454fa5cf0877c91f86ed as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/9e24d40cf4cf454fa5cf0877c91f86ed 2023-04-20 12:54:00,831 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/9e24d40cf4cf454fa5cf0877c91f86ed, entries=150, sequenceid=923, filesize=11.9 K 2023-04-20 12:54:00,832 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/B/215e7857cc9842f6ab66f8c82cec3980 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/215e7857cc9842f6ab66f8c82cec3980 2023-04-20 12:54:00,839 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/215e7857cc9842f6ab66f8c82cec3980, entries=150, sequenceid=923, filesize=11.9 K 2023-04-20 12:54:00,841 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/.tmp/C/06d6a20e2c224230b701587e49d78cbf as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/06d6a20e2c224230b701587e49d78cbf 2023-04-20 12:54:00,846 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/06d6a20e2c224230b701587e49d78cbf, entries=150, sequenceid=923, filesize=11.9 K 2023-04-20 12:54:00,847 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for fc3812e05598851193449f5684adf036 in 1289ms, sequenceid=923, compaction requested=true 2023-04-20 12:54:00,868 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/4380c2629118436596ef388bf4a5ec79, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/178bcaefe7ff4cfdbe8f346159abe564, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/29269cc720484c778e35bcbdcb14c9b1, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/114a10a003ef498880821855ded8127c, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/dc2f827330d94cd9a578332dd10a5e73, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/767c75e9bff84f358366527ace38efb8, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/15bd28d10e954c4d96a60c94428f59d5, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/b169c2caecc4494bafed10149a40d74d, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/d2510056be1846ef912b783cbf62405e, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/536f4132bba14b79b059d6146e90f070, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/0c1b15029a144d9d98f5a823934c5ba3, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/103039a0ca884bf39bd298e171bb47b9, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/8932caa62d6544a39c1bc2521de7ebc7, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/bb63fee100424f148013afe63ee26fcd, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/4a426bcc9c964cc49ed1465eaa014840, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/0297843f356f479cba7ba7e6449b8894, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/d80b97f269514bbb8233513d9e628405, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/35b85ba6987449c284859e423ea2557b, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/cb98bb476a524fd2b78738a30d5756c3, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/a5a4eea6622a498382819d352fa3bacd, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/6b5d4f0595a9498c87a9a25c6b74ac78, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/6759b824d89340a09fd4a2208ddcedb9, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/9e248d37c9fd42b6ad13bd2faad13cec, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/7f9286b9a9db424ebda6519b8c85ba84, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/01426621474b40eab00989d3de456745, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/3a6988e4560d47e6829169d3dedd7569, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/e128a06d96634e8bb6e57329d757fe0e, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/ff28ec31a4e24e23b15756a037f0d896, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/8d378d1628394ae78de516138f4050f4, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/ad240b4ba20a4298bf062b4de2148a45, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/6e6beeb295d84d94a3c6ebc09e6e12f4, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/cb84af0e3ea34068aedde842aaecbec2, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/836e0a83f4984fc49c00f471c84b17aa] to archive 2023-04-20 12:54:00,869 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-20 12:54:00,874 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/4380c2629118436596ef388bf4a5ec79 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/4380c2629118436596ef388bf4a5ec79 2023-04-20 12:54:00,876 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/178bcaefe7ff4cfdbe8f346159abe564 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/178bcaefe7ff4cfdbe8f346159abe564 2023-04-20 12:54:00,877 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/29269cc720484c778e35bcbdcb14c9b1 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/29269cc720484c778e35bcbdcb14c9b1 2023-04-20 12:54:00,878 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/114a10a003ef498880821855ded8127c to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/114a10a003ef498880821855ded8127c 2023-04-20 12:54:00,879 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/dc2f827330d94cd9a578332dd10a5e73 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/dc2f827330d94cd9a578332dd10a5e73 2023-04-20 12:54:00,881 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/767c75e9bff84f358366527ace38efb8 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/767c75e9bff84f358366527ace38efb8 2023-04-20 12:54:00,882 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/15bd28d10e954c4d96a60c94428f59d5 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/15bd28d10e954c4d96a60c94428f59d5 2023-04-20 12:54:00,883 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/b169c2caecc4494bafed10149a40d74d to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/b169c2caecc4494bafed10149a40d74d 2023-04-20 12:54:00,884 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/d2510056be1846ef912b783cbf62405e to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/d2510056be1846ef912b783cbf62405e 2023-04-20 12:54:00,885 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/536f4132bba14b79b059d6146e90f070 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/536f4132bba14b79b059d6146e90f070 2023-04-20 12:54:00,886 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/0c1b15029a144d9d98f5a823934c5ba3 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/0c1b15029a144d9d98f5a823934c5ba3 2023-04-20 12:54:00,887 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/103039a0ca884bf39bd298e171bb47b9 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/103039a0ca884bf39bd298e171bb47b9 2023-04-20 12:54:00,888 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/8932caa62d6544a39c1bc2521de7ebc7 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/8932caa62d6544a39c1bc2521de7ebc7 2023-04-20 12:54:00,889 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/bb63fee100424f148013afe63ee26fcd to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/bb63fee100424f148013afe63ee26fcd 2023-04-20 12:54:00,890 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/4a426bcc9c964cc49ed1465eaa014840 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/4a426bcc9c964cc49ed1465eaa014840 2023-04-20 12:54:00,891 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/0297843f356f479cba7ba7e6449b8894 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/0297843f356f479cba7ba7e6449b8894 2023-04-20 12:54:00,892 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/d80b97f269514bbb8233513d9e628405 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/d80b97f269514bbb8233513d9e628405 2023-04-20 12:54:00,893 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/35b85ba6987449c284859e423ea2557b to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/35b85ba6987449c284859e423ea2557b 2023-04-20 12:54:00,893 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/cb98bb476a524fd2b78738a30d5756c3 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/cb98bb476a524fd2b78738a30d5756c3 2023-04-20 12:54:00,895 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/a5a4eea6622a498382819d352fa3bacd to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/a5a4eea6622a498382819d352fa3bacd 2023-04-20 12:54:00,896 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/6b5d4f0595a9498c87a9a25c6b74ac78 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/6b5d4f0595a9498c87a9a25c6b74ac78 2023-04-20 12:54:00,896 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/6759b824d89340a09fd4a2208ddcedb9 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/6759b824d89340a09fd4a2208ddcedb9 2023-04-20 12:54:00,897 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/9e248d37c9fd42b6ad13bd2faad13cec to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/9e248d37c9fd42b6ad13bd2faad13cec 2023-04-20 12:54:00,898 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/7f9286b9a9db424ebda6519b8c85ba84 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/7f9286b9a9db424ebda6519b8c85ba84 2023-04-20 12:54:00,899 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/01426621474b40eab00989d3de456745 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/01426621474b40eab00989d3de456745 2023-04-20 12:54:00,900 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/3a6988e4560d47e6829169d3dedd7569 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/3a6988e4560d47e6829169d3dedd7569 2023-04-20 12:54:00,901 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/e128a06d96634e8bb6e57329d757fe0e to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/e128a06d96634e8bb6e57329d757fe0e 2023-04-20 12:54:00,902 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/ff28ec31a4e24e23b15756a037f0d896 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/ff28ec31a4e24e23b15756a037f0d896 2023-04-20 12:54:00,902 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/8d378d1628394ae78de516138f4050f4 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/8d378d1628394ae78de516138f4050f4 2023-04-20 12:54:00,903 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/ad240b4ba20a4298bf062b4de2148a45 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/ad240b4ba20a4298bf062b4de2148a45 2023-04-20 12:54:00,905 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/6e6beeb295d84d94a3c6ebc09e6e12f4 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/6e6beeb295d84d94a3c6ebc09e6e12f4 2023-04-20 12:54:00,906 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/cb84af0e3ea34068aedde842aaecbec2 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/cb84af0e3ea34068aedde842aaecbec2 2023-04-20 12:54:00,907 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/836e0a83f4984fc49c00f471c84b17aa to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/836e0a83f4984fc49c00f471c84b17aa 2023-04-20 12:54:00,931 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/fd256aaf01fc45cf92a27550dee35a51, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/6087b0ef524e4d73a5434f8887e20acf, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/43ef446734d849da8b24680e7f86d638, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/f856402354ad47a9ae6b6db58bbcd048, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/855f3a218b3f457a9578927285d93556, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/d303006629324fbcad8537dd27e10df4, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/9e910ba6e49f4ad5ad677c7837d4902a, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/b2527240d2ca4f2f8d735a88d4aa7835, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/51f1b23a2d8f4dde9c1879ea1c8bf37f, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/6c6ab749f3ad4ba79733dffe591843d8, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/45eb80f0dae9487ba746bd611d21bd6a, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/ce4a2b24f0ff4e568b435ac022ab24e2, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/f6e05bfe8ed74791804c16123ac6ad79, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/a15ffd1b73414cce82dd63db6a945b67, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/7264c8e180ac4b20b5d64aecc562ad61, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/dfcb85b67759440f8d1526b85663e3cc, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/f38625ab486645f3bfa7076eb9145abe, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/b9bccce7e7b14ca98390dd4f58931c86, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/d8642be750364a77a8c5e8afe1a9b82f, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/b8fda890af564f37bbd57bb07d32d493, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/42ceec74dd204cf2a28b6ac3054967b8, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/a214f6ecfda44a83a3bffc62c428aca7, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/ad789420b4104b96a940eab738cb097d, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/23d0442755e640599fd3ff90ce861c42, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/5bb9db7c1cd842ef87f1cfed6e7ef1f0, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/f59fcb4473e642269925310c4701c745, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/d2b7d80bafa0416dbf090160c5411bfe, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/dad322557f66472ea42e0df4a925f516, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/0a10eb7e252541eba72194f18802b833, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/71aca374bec44b81989b14843390a16b, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/3f9d1036026744e983641ed2753ba955, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/4b45f240e3c242f58ba1378e6ceac8c3, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/ebb5e025f03b44a9b9cf95138aceff8b] to archive 2023-04-20 12:54:00,933 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-20 12:54:00,935 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/fd256aaf01fc45cf92a27550dee35a51 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/fd256aaf01fc45cf92a27550dee35a51 2023-04-20 12:54:00,936 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/6087b0ef524e4d73a5434f8887e20acf to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/6087b0ef524e4d73a5434f8887e20acf 2023-04-20 12:54:00,937 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/43ef446734d849da8b24680e7f86d638 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/43ef446734d849da8b24680e7f86d638 2023-04-20 12:54:00,938 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/f856402354ad47a9ae6b6db58bbcd048 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/f856402354ad47a9ae6b6db58bbcd048 2023-04-20 12:54:00,940 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/855f3a218b3f457a9578927285d93556 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/855f3a218b3f457a9578927285d93556 2023-04-20 12:54:00,941 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/d303006629324fbcad8537dd27e10df4 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/d303006629324fbcad8537dd27e10df4 2023-04-20 12:54:00,942 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/9e910ba6e49f4ad5ad677c7837d4902a to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/9e910ba6e49f4ad5ad677c7837d4902a 2023-04-20 12:54:00,943 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/b2527240d2ca4f2f8d735a88d4aa7835 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/b2527240d2ca4f2f8d735a88d4aa7835 2023-04-20 12:54:00,944 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/51f1b23a2d8f4dde9c1879ea1c8bf37f to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/51f1b23a2d8f4dde9c1879ea1c8bf37f 2023-04-20 12:54:00,945 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/6c6ab749f3ad4ba79733dffe591843d8 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/6c6ab749f3ad4ba79733dffe591843d8 2023-04-20 12:54:00,946 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/45eb80f0dae9487ba746bd611d21bd6a to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/45eb80f0dae9487ba746bd611d21bd6a 2023-04-20 12:54:00,947 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/ce4a2b24f0ff4e568b435ac022ab24e2 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/ce4a2b24f0ff4e568b435ac022ab24e2 2023-04-20 12:54:00,949 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/f6e05bfe8ed74791804c16123ac6ad79 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/f6e05bfe8ed74791804c16123ac6ad79 2023-04-20 12:54:00,950 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/a15ffd1b73414cce82dd63db6a945b67 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/a15ffd1b73414cce82dd63db6a945b67 2023-04-20 12:54:00,952 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/7264c8e180ac4b20b5d64aecc562ad61 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/7264c8e180ac4b20b5d64aecc562ad61 2023-04-20 12:54:00,953 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/dfcb85b67759440f8d1526b85663e3cc to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/dfcb85b67759440f8d1526b85663e3cc 2023-04-20 12:54:00,955 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/f38625ab486645f3bfa7076eb9145abe to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/f38625ab486645f3bfa7076eb9145abe 2023-04-20 12:54:00,956 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/b9bccce7e7b14ca98390dd4f58931c86 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/b9bccce7e7b14ca98390dd4f58931c86 2023-04-20 12:54:00,958 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/d8642be750364a77a8c5e8afe1a9b82f to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/d8642be750364a77a8c5e8afe1a9b82f 2023-04-20 12:54:00,960 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/b8fda890af564f37bbd57bb07d32d493 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/b8fda890af564f37bbd57bb07d32d493 2023-04-20 12:54:00,961 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/42ceec74dd204cf2a28b6ac3054967b8 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/42ceec74dd204cf2a28b6ac3054967b8 2023-04-20 12:54:00,963 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/a214f6ecfda44a83a3bffc62c428aca7 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/a214f6ecfda44a83a3bffc62c428aca7 2023-04-20 12:54:00,964 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/ad789420b4104b96a940eab738cb097d to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/ad789420b4104b96a940eab738cb097d 2023-04-20 12:54:00,966 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/23d0442755e640599fd3ff90ce861c42 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/23d0442755e640599fd3ff90ce861c42 2023-04-20 12:54:00,967 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/5bb9db7c1cd842ef87f1cfed6e7ef1f0 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/5bb9db7c1cd842ef87f1cfed6e7ef1f0 2023-04-20 12:54:00,968 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/f59fcb4473e642269925310c4701c745 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/f59fcb4473e642269925310c4701c745 2023-04-20 12:54:00,970 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/d2b7d80bafa0416dbf090160c5411bfe to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/d2b7d80bafa0416dbf090160c5411bfe 2023-04-20 12:54:00,971 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/dad322557f66472ea42e0df4a925f516 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/dad322557f66472ea42e0df4a925f516 2023-04-20 12:54:00,972 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/0a10eb7e252541eba72194f18802b833 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/0a10eb7e252541eba72194f18802b833 2023-04-20 12:54:00,973 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/71aca374bec44b81989b14843390a16b to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/71aca374bec44b81989b14843390a16b 2023-04-20 12:54:00,975 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/3f9d1036026744e983641ed2753ba955 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/3f9d1036026744e983641ed2753ba955 2023-04-20 12:54:00,976 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/4b45f240e3c242f58ba1378e6ceac8c3 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/4b45f240e3c242f58ba1378e6ceac8c3 2023-04-20 12:54:00,977 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/ebb5e025f03b44a9b9cf95138aceff8b to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/ebb5e025f03b44a9b9cf95138aceff8b 2023-04-20 12:54:00,981 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/04f4bc85503742678ba998a28f08db27, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/4517f1c958094b5aae739a01a3af2e14, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/57ef34ba05054f47b630839ed7c7b41e, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/0b26314a595f497cad9b2f78ecc05890, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/4d34e0a359ac4248804192649298ce59, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/4d56472ab5dc43c7a7945e2605ec59ca, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/523c7f28b3bd476e8d3bf0dcfa8ba35f, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/254fbb5994d74ac1bd4886aa60a4f25c, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/8cf2ab1f6be6404fb8aa879543aba93e, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/019aa894e9a541b4b205239822ff57de, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/c8376418c06840e28cd81a1b5668fe60, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/78ccd2a3ffb043f0b7765caeb6cb1b5a, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/88e09a6207ea48948d108830316f4167, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/f5b7df47a1ac42938efc566203740c93, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/c638e4cbfb4141f281f070ea1885ae0f, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/3e1faa50ac584b8abda498bb572312b2, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/10c83a7919ff438fb1e051e13ff01a1d, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/89cc93c45b4445dc8875cea52b50ad56, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/ced07e9101ae4c429c0b773207715999, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/a4780354b5ad4a9d93e998a640badd1c, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/2fc2928bc712431c8de9a78335e480b6, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/69e6ebe7b57e4062b788b93869f71b2f, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/ac914b06475449d49bacd87ca89646ea, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/2118ba1f99a249a58d3f13969814c1cd, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/0e9f011979044e8fbbbb1d94703a3890, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/5671d40814264aebbe71aa0da47beb00, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/667275278d6c4cf89f592bb40992dbe9, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/f366ed4667864bab8f6942927760a4f9, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/3426a67328f94f3aa6bd60deba73f639, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/30d4f66f1c9f4fb6a8667aa094238960, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/fa3b99aa76cf4696a9eb4871db878a0d] to archive 2023-04-20 12:54:00,982 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-20 12:54:00,983 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/04f4bc85503742678ba998a28f08db27 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/04f4bc85503742678ba998a28f08db27 2023-04-20 12:54:00,984 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/4517f1c958094b5aae739a01a3af2e14 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/4517f1c958094b5aae739a01a3af2e14 2023-04-20 12:54:00,986 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/57ef34ba05054f47b630839ed7c7b41e to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/57ef34ba05054f47b630839ed7c7b41e 2023-04-20 12:54:00,987 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/0b26314a595f497cad9b2f78ecc05890 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/0b26314a595f497cad9b2f78ecc05890 2023-04-20 12:54:00,988 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/4d34e0a359ac4248804192649298ce59 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/4d34e0a359ac4248804192649298ce59 2023-04-20 12:54:00,989 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/4d56472ab5dc43c7a7945e2605ec59ca to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/4d56472ab5dc43c7a7945e2605ec59ca 2023-04-20 12:54:00,990 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/523c7f28b3bd476e8d3bf0dcfa8ba35f to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/523c7f28b3bd476e8d3bf0dcfa8ba35f 2023-04-20 12:54:00,991 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/254fbb5994d74ac1bd4886aa60a4f25c to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/254fbb5994d74ac1bd4886aa60a4f25c 2023-04-20 12:54:00,992 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/8cf2ab1f6be6404fb8aa879543aba93e to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/8cf2ab1f6be6404fb8aa879543aba93e 2023-04-20 12:54:00,993 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/019aa894e9a541b4b205239822ff57de to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/019aa894e9a541b4b205239822ff57de 2023-04-20 12:54:00,995 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/c8376418c06840e28cd81a1b5668fe60 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/c8376418c06840e28cd81a1b5668fe60 2023-04-20 12:54:00,996 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/78ccd2a3ffb043f0b7765caeb6cb1b5a to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/78ccd2a3ffb043f0b7765caeb6cb1b5a 2023-04-20 12:54:00,997 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/88e09a6207ea48948d108830316f4167 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/88e09a6207ea48948d108830316f4167 2023-04-20 12:54:00,998 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/f5b7df47a1ac42938efc566203740c93 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/f5b7df47a1ac42938efc566203740c93 2023-04-20 12:54:01,000 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/c638e4cbfb4141f281f070ea1885ae0f to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/c638e4cbfb4141f281f070ea1885ae0f 2023-04-20 12:54:01,001 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/3e1faa50ac584b8abda498bb572312b2 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/3e1faa50ac584b8abda498bb572312b2 2023-04-20 12:54:01,002 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/10c83a7919ff438fb1e051e13ff01a1d to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/10c83a7919ff438fb1e051e13ff01a1d 2023-04-20 12:54:01,003 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/89cc93c45b4445dc8875cea52b50ad56 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/89cc93c45b4445dc8875cea52b50ad56 2023-04-20 12:54:01,004 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/ced07e9101ae4c429c0b773207715999 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/ced07e9101ae4c429c0b773207715999 2023-04-20 12:54:01,005 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/a4780354b5ad4a9d93e998a640badd1c to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/a4780354b5ad4a9d93e998a640badd1c 2023-04-20 12:54:01,006 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/2fc2928bc712431c8de9a78335e480b6 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/2fc2928bc712431c8de9a78335e480b6 2023-04-20 12:54:01,007 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/69e6ebe7b57e4062b788b93869f71b2f to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/69e6ebe7b57e4062b788b93869f71b2f 2023-04-20 12:54:01,008 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/ac914b06475449d49bacd87ca89646ea to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/ac914b06475449d49bacd87ca89646ea 2023-04-20 12:54:01,009 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/2118ba1f99a249a58d3f13969814c1cd to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/2118ba1f99a249a58d3f13969814c1cd 2023-04-20 12:54:01,011 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/0e9f011979044e8fbbbb1d94703a3890 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/0e9f011979044e8fbbbb1d94703a3890 2023-04-20 12:54:01,012 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/5671d40814264aebbe71aa0da47beb00 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/5671d40814264aebbe71aa0da47beb00 2023-04-20 12:54:01,013 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/667275278d6c4cf89f592bb40992dbe9 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/667275278d6c4cf89f592bb40992dbe9 2023-04-20 12:54:01,014 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/f366ed4667864bab8f6942927760a4f9 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/f366ed4667864bab8f6942927760a4f9 2023-04-20 12:54:01,015 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/3426a67328f94f3aa6bd60deba73f639 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/3426a67328f94f3aa6bd60deba73f639 2023-04-20 12:54:01,016 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/30d4f66f1c9f4fb6a8667aa094238960 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/30d4f66f1c9f4fb6a8667aa094238960 2023-04-20 12:54:01,017 DEBUG [StoreCloser-TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/fa3b99aa76cf4696a9eb4871db878a0d to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/fa3b99aa76cf4696a9eb4871db878a0d 2023-04-20 12:54:01,021 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/recovered.edits/926.seqid, newMaxSeqId=926, maxSeqId=1 2023-04-20 12:54:01,022 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036. 2023-04-20 12:54:01,022 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for fc3812e05598851193449f5684adf036: 2023-04-20 12:54:01,024 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed fc3812e05598851193449f5684adf036 2023-04-20 12:54:01,025 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=fc3812e05598851193449f5684adf036, regionState=CLOSED 2023-04-20 12:54:01,025 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681995241025"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681995241025"}]},"ts":"1681995241025"} 2023-04-20 12:54:01,027 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-04-20 12:54:01,027 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure fc3812e05598851193449f5684adf036, server=jenkins-hbase20.apache.org,36277,1681995213947 in 1.6290 sec 2023-04-20 12:54:01,028 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-04-20 12:54:01,028 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=fc3812e05598851193449f5684adf036, UNASSIGN in 1.6330 sec 2023-04-20 12:54:01,029 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681995241029"}]},"ts":"1681995241029"} 2023-04-20 12:54:01,030 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-20 12:54:01,031 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-20 12:54:01,033 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.6500 sec 2023-04-20 12:54:01,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-20 12:54:01,498 INFO [Listener at localhost.localdomain/42205] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-04-20 12:54:01,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-20 12:54:01,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 12:54:01,516 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 12:54:01,518 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 12:54:01,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-20 12:54:01,524 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036 2023-04-20 12:54:01,528 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A, FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B, FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C, FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/recovered.edits] 2023-04-20 12:54:01,531 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/5534fdf5991b45b599c287a9384ebc08 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/5534fdf5991b45b599c287a9384ebc08 2023-04-20 12:54:01,533 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/656af3ca25f04368847b3f151a85fe03 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/656af3ca25f04368847b3f151a85fe03 2023-04-20 12:54:01,534 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/9e24d40cf4cf454fa5cf0877c91f86ed to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/A/9e24d40cf4cf454fa5cf0877c91f86ed 2023-04-20 12:54:01,537 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/215e7857cc9842f6ab66f8c82cec3980 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/215e7857cc9842f6ab66f8c82cec3980 2023-04-20 12:54:01,538 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/d9c663aa7fc34ec4b682c1378cb95025 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/d9c663aa7fc34ec4b682c1378cb95025 2023-04-20 12:54:01,539 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/ea28721b65ac4205adb4ba5e5f566f8b to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/B/ea28721b65ac4205adb4ba5e5f566f8b 2023-04-20 12:54:01,542 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/06d6a20e2c224230b701587e49d78cbf to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/06d6a20e2c224230b701587e49d78cbf 2023-04-20 12:54:01,543 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/369ddf384c484c0b89c4293fb7b22001 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/369ddf384c484c0b89c4293fb7b22001 2023-04-20 12:54:01,544 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/780153d01a5c4f4fa3126bbe3efcbf12 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/C/780153d01a5c4f4fa3126bbe3efcbf12 2023-04-20 12:54:01,547 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/recovered.edits/926.seqid to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036/recovered.edits/926.seqid 2023-04-20 12:54:01,548 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/fc3812e05598851193449f5684adf036 2023-04-20 12:54:01,548 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-20 12:54:01,551 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 12:54:01,575 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-20 12:54:01,612 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-20 12:54:01,614 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 12:54:01,614 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-20 12:54:01,614 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681995241614"}]},"ts":"9223372036854775807"} 2023-04-20 12:54:01,616 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-20 12:54:01,616 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => fc3812e05598851193449f5684adf036, NAME => 'TestAcidGuarantees,,1681995217281.fc3812e05598851193449f5684adf036.', STARTKEY => '', ENDKEY => ''}] 2023-04-20 12:54:01,616 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-20 12:54:01,616 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681995241616"}]},"ts":"9223372036854775807"} 2023-04-20 12:54:01,618 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-20 12:54:01,619 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 12:54:01,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-20 12:54:01,621 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 113 msec 2023-04-20 12:54:01,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-20 12:54:01,822 INFO [Listener at localhost.localdomain/42205] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-04-20 12:54:01,840 INFO [Listener at localhost.localdomain/42205] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=236 (was 207) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1938098848_15 at /127.0.0.1:40098 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x58166cba-shared-pool-20 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1938098848_15 at /127.0.0.1:43806 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase20:36277-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1938098848_15 at /127.0.0.1:43752 [Waiting for operation #7] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x58166cba-shared-pool-21 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x58166cba-shared-pool-19 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1938098848_15 at /127.0.0.1:43708 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x58166cba-shared-pool-22 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=559 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=344 (was 298) - SystemLoadAverage LEAK? -, ProcessCount=172 (was 172), AvailableMemoryMB=9312 (was 9803) 2023-04-20 12:54:01,851 INFO [Listener at localhost.localdomain/42205] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=236, OpenFileDescriptor=559, MaxFileDescriptor=60000, SystemLoadAverage=344, ProcessCount=172, AvailableMemoryMB=9309 2023-04-20 12:54:01,853 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-20 12:54:01,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-20 12:54:01,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-20 12:54:01,857 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-20 12:54:01,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-04-20 12:54:01,858 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-20 12:54:01,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-20 12:54:01,861 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953 2023-04-20 12:54:01,862 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953 empty. 2023-04-20 12:54:01,862 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953 2023-04-20 12:54:01,862 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-20 12:54:01,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-20 12:54:02,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-20 12:54:02,274 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-20 12:54:02,276 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 4782eca030b67d2c82cb20827b0bb953, NAME => 'TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp 2023-04-20 12:54:02,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-20 12:54:02,685 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-20 12:54:02,685 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 4782eca030b67d2c82cb20827b0bb953, disabling compactions & flushes 2023-04-20 12:54:02,685 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:02,685 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:02,685 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. after waiting 0 ms 2023-04-20 12:54:02,685 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:02,685 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:02,685 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 4782eca030b67d2c82cb20827b0bb953: 2023-04-20 12:54:02,690 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-20 12:54:02,693 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681995242692"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681995242692"}]},"ts":"1681995242692"} 2023-04-20 12:54:02,695 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-20 12:54:02,697 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-20 12:54:02,697 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681995242697"}]},"ts":"1681995242697"} 2023-04-20 12:54:02,698 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-20 12:54:02,700 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4782eca030b67d2c82cb20827b0bb953, ASSIGN}] 2023-04-20 12:54:02,702 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4782eca030b67d2c82cb20827b0bb953, ASSIGN 2023-04-20 12:54:02,703 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=4782eca030b67d2c82cb20827b0bb953, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,36277,1681995213947; forceNewPlan=false, retain=false 2023-04-20 12:54:02,855 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=4782eca030b67d2c82cb20827b0bb953, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:02,856 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681995242855"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681995242855"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681995242855"}]},"ts":"1681995242855"} 2023-04-20 12:54:02,859 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure 4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947}] 2023-04-20 12:54:02,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-20 12:54:03,016 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:03,016 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 4782eca030b67d2c82cb20827b0bb953, NAME => 'TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.', STARTKEY => '', ENDKEY => ''} 2023-04-20 12:54:03,017 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 4782eca030b67d2c82cb20827b0bb953 2023-04-20 12:54:03,017 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-20 12:54:03,017 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 4782eca030b67d2c82cb20827b0bb953 2023-04-20 12:54:03,017 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 4782eca030b67d2c82cb20827b0bb953 2023-04-20 12:54:03,018 INFO [StoreOpener-4782eca030b67d2c82cb20827b0bb953-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 4782eca030b67d2c82cb20827b0bb953 2023-04-20 12:54:03,020 DEBUG [StoreOpener-4782eca030b67d2c82cb20827b0bb953-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A 2023-04-20 12:54:03,020 DEBUG [StoreOpener-4782eca030b67d2c82cb20827b0bb953-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A 2023-04-20 12:54:03,020 INFO [StoreOpener-4782eca030b67d2c82cb20827b0bb953-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 12:54:03,020 INFO [StoreOpener-4782eca030b67d2c82cb20827b0bb953-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4782eca030b67d2c82cb20827b0bb953 columnFamilyName A 2023-04-20 12:54:03,020 INFO [StoreOpener-4782eca030b67d2c82cb20827b0bb953-1] regionserver.HStore(310): Store=4782eca030b67d2c82cb20827b0bb953/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 12:54:03,021 INFO [StoreOpener-4782eca030b67d2c82cb20827b0bb953-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 4782eca030b67d2c82cb20827b0bb953 2023-04-20 12:54:03,022 DEBUG [StoreOpener-4782eca030b67d2c82cb20827b0bb953-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B 2023-04-20 12:54:03,022 DEBUG [StoreOpener-4782eca030b67d2c82cb20827b0bb953-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B 2023-04-20 12:54:03,022 INFO [StoreOpener-4782eca030b67d2c82cb20827b0bb953-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 12:54:03,022 INFO [StoreOpener-4782eca030b67d2c82cb20827b0bb953-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4782eca030b67d2c82cb20827b0bb953 columnFamilyName B 2023-04-20 12:54:03,023 INFO [StoreOpener-4782eca030b67d2c82cb20827b0bb953-1] regionserver.HStore(310): Store=4782eca030b67d2c82cb20827b0bb953/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 12:54:03,023 INFO [StoreOpener-4782eca030b67d2c82cb20827b0bb953-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 4782eca030b67d2c82cb20827b0bb953 2023-04-20 12:54:03,024 DEBUG [StoreOpener-4782eca030b67d2c82cb20827b0bb953-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C 2023-04-20 12:54:03,024 DEBUG [StoreOpener-4782eca030b67d2c82cb20827b0bb953-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C 2023-04-20 12:54:03,024 INFO [StoreOpener-4782eca030b67d2c82cb20827b0bb953-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 12:54:03,024 INFO [StoreOpener-4782eca030b67d2c82cb20827b0bb953-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4782eca030b67d2c82cb20827b0bb953 columnFamilyName C 2023-04-20 12:54:03,025 INFO [StoreOpener-4782eca030b67d2c82cb20827b0bb953-1] regionserver.HStore(310): Store=4782eca030b67d2c82cb20827b0bb953/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 12:54:03,025 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:03,025 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953 2023-04-20 12:54:03,026 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953 2023-04-20 12:54:03,028 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-20 12:54:03,029 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 4782eca030b67d2c82cb20827b0bb953 2023-04-20 12:54:03,031 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-20 12:54:03,031 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 4782eca030b67d2c82cb20827b0bb953; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64582907, jitterRate=-0.037639692425727844}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-20 12:54:03,032 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 4782eca030b67d2c82cb20827b0bb953: 2023-04-20 12:54:03,032 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953., pid=18, masterSystemTime=1681995243013 2023-04-20 12:54:03,034 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:03,034 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:03,034 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=4782eca030b67d2c82cb20827b0bb953, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:03,035 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681995243034"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1681995243034"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681995243034"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681995243034"}]},"ts":"1681995243034"} 2023-04-20 12:54:03,038 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-04-20 12:54:03,039 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure 4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 in 177 msec 2023-04-20 12:54:03,041 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-04-20 12:54:03,041 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=4782eca030b67d2c82cb20827b0bb953, ASSIGN in 339 msec 2023-04-20 12:54:03,042 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-20 12:54:03,042 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681995243042"}]},"ts":"1681995243042"} 2023-04-20 12:54:03,043 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-20 12:54:03,045 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-20 12:54:03,046 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1910 sec 2023-04-20 12:54:03,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-20 12:54:03,968 INFO [Listener at localhost.localdomain/42205] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-04-20 12:54:03,969 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x10dd04e6 to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:54:03,978 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4258ab17, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:54:03,981 DEBUG [Listener at localhost.localdomain/42205] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:54:03,985 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34188, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:54:03,987 DEBUG [Listener at localhost.localdomain/42205] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-20 12:54:03,991 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:45044, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-20 12:54:03,998 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-20 12:54:03,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.HMaster$12(2454): Client=jenkins//148.251.75.209 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-20 12:54:04,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-20 12:54:04,422 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-20 12:54:04,424 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-20 12:54:04,424 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-20 12:54:04,430 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-20 12:54:04,442 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4782eca030b67d2c82cb20827b0bb953, REOPEN/MOVE}] 2023-04-20 12:54:04,443 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4782eca030b67d2c82cb20827b0bb953, REOPEN/MOVE 2023-04-20 12:54:04,444 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=4782eca030b67d2c82cb20827b0bb953, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:04,444 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681995244444"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681995244444"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681995244444"}]},"ts":"1681995244444"} 2023-04-20 12:54:04,445 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure 4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947}] 2023-04-20 12:54:04,597 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 4782eca030b67d2c82cb20827b0bb953 2023-04-20 12:54:04,598 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 4782eca030b67d2c82cb20827b0bb953, disabling compactions & flushes 2023-04-20 12:54:04,598 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:04,598 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:04,598 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. after waiting 0 ms 2023-04-20 12:54:04,598 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:04,602 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-20 12:54:04,603 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:04,603 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 4782eca030b67d2c82cb20827b0bb953: 2023-04-20 12:54:04,603 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 4782eca030b67d2c82cb20827b0bb953 to self. 2023-04-20 12:54:04,605 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 4782eca030b67d2c82cb20827b0bb953 2023-04-20 12:54:04,606 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=4782eca030b67d2c82cb20827b0bb953, regionState=CLOSED 2023-04-20 12:54:04,606 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681995244606"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681995244606"}]},"ts":"1681995244606"} 2023-04-20 12:54:04,610 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-04-20 12:54:04,610 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure 4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 in 163 msec 2023-04-20 12:54:04,611 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=4782eca030b67d2c82cb20827b0bb953, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase20.apache.org,36277,1681995213947; forceNewPlan=false, retain=true 2023-04-20 12:54:04,762 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=4782eca030b67d2c82cb20827b0bb953, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:04,762 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681995244762"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681995244762"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681995244762"}]},"ts":"1681995244762"} 2023-04-20 12:54:04,766 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure 4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947}] 2023-04-20 12:54:04,923 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:04,923 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 4782eca030b67d2c82cb20827b0bb953, NAME => 'TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.', STARTKEY => '', ENDKEY => ''} 2023-04-20 12:54:04,924 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 4782eca030b67d2c82cb20827b0bb953 2023-04-20 12:54:04,924 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-20 12:54:04,924 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 4782eca030b67d2c82cb20827b0bb953 2023-04-20 12:54:04,924 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 4782eca030b67d2c82cb20827b0bb953 2023-04-20 12:54:04,927 INFO [StoreOpener-4782eca030b67d2c82cb20827b0bb953-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 4782eca030b67d2c82cb20827b0bb953 2023-04-20 12:54:04,929 DEBUG [StoreOpener-4782eca030b67d2c82cb20827b0bb953-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A 2023-04-20 12:54:04,929 DEBUG [StoreOpener-4782eca030b67d2c82cb20827b0bb953-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A 2023-04-20 12:54:04,929 INFO [StoreOpener-4782eca030b67d2c82cb20827b0bb953-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 12:54:04,932 INFO [StoreOpener-4782eca030b67d2c82cb20827b0bb953-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4782eca030b67d2c82cb20827b0bb953 columnFamilyName A 2023-04-20 12:54:04,935 INFO [StoreOpener-4782eca030b67d2c82cb20827b0bb953-1] regionserver.HStore(310): Store=4782eca030b67d2c82cb20827b0bb953/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 12:54:04,935 INFO [StoreOpener-4782eca030b67d2c82cb20827b0bb953-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 4782eca030b67d2c82cb20827b0bb953 2023-04-20 12:54:04,937 DEBUG [StoreOpener-4782eca030b67d2c82cb20827b0bb953-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B 2023-04-20 12:54:04,937 DEBUG [StoreOpener-4782eca030b67d2c82cb20827b0bb953-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B 2023-04-20 12:54:04,937 INFO [StoreOpener-4782eca030b67d2c82cb20827b0bb953-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 12:54:04,937 INFO [StoreOpener-4782eca030b67d2c82cb20827b0bb953-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4782eca030b67d2c82cb20827b0bb953 columnFamilyName B 2023-04-20 12:54:04,938 INFO [StoreOpener-4782eca030b67d2c82cb20827b0bb953-1] regionserver.HStore(310): Store=4782eca030b67d2c82cb20827b0bb953/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 12:54:04,938 INFO [StoreOpener-4782eca030b67d2c82cb20827b0bb953-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 4782eca030b67d2c82cb20827b0bb953 2023-04-20 12:54:04,940 DEBUG [StoreOpener-4782eca030b67d2c82cb20827b0bb953-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C 2023-04-20 12:54:04,940 DEBUG [StoreOpener-4782eca030b67d2c82cb20827b0bb953-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C 2023-04-20 12:54:04,940 INFO [StoreOpener-4782eca030b67d2c82cb20827b0bb953-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 12:54:04,941 INFO [StoreOpener-4782eca030b67d2c82cb20827b0bb953-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4782eca030b67d2c82cb20827b0bb953 columnFamilyName C 2023-04-20 12:54:04,941 INFO [StoreOpener-4782eca030b67d2c82cb20827b0bb953-1] regionserver.HStore(310): Store=4782eca030b67d2c82cb20827b0bb953/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 12:54:04,942 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:04,942 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953 2023-04-20 12:54:04,944 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953 2023-04-20 12:54:04,946 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-20 12:54:04,947 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 4782eca030b67d2c82cb20827b0bb953 2023-04-20 12:54:04,948 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 4782eca030b67d2c82cb20827b0bb953; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61839232, jitterRate=-0.07852363586425781}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-20 12:54:04,948 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 4782eca030b67d2c82cb20827b0bb953: 2023-04-20 12:54:04,948 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953., pid=23, masterSystemTime=1681995244919 2023-04-20 12:54:04,950 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:04,950 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:04,950 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=4782eca030b67d2c82cb20827b0bb953, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:04,951 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681995244950"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1681995244950"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681995244950"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681995244950"}]},"ts":"1681995244950"} 2023-04-20 12:54:04,953 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-04-20 12:54:04,953 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure 4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 in 186 msec 2023-04-20 12:54:04,954 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-04-20 12:54:04,955 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=4782eca030b67d2c82cb20827b0bb953, REOPEN/MOVE in 511 msec 2023-04-20 12:54:04,957 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-04-20 12:54:04,957 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 528 msec 2023-04-20 12:54:04,959 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 957 msec 2023-04-20 12:54:04,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-20 12:54:04,970 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x6dc16927 to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:54:04,975 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@65dbf2b7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:54:04,976 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x029d68f7 to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:54:04,981 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@79385b7c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:54:04,981 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x6322d500 to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:54:04,986 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@a11c343, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:54:04,987 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x4af56759 to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:54:05,002 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4e7aae9f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:54:05,002 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x5d07b2f5 to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:54:05,014 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3ca9d9a3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:54:05,014 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x34b669b1 to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:54:05,022 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@46a0ca45, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:54:05,023 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x6683e622 to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:54:05,027 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@61fff0a2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:54:05,027 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x615d8f34 to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:54:05,032 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1896d7cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:54:05,032 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x4890ce53 to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:54:05,039 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@45b9cc5a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:54:05,048 DEBUG [hconnection-0x8fa65d8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:54:05,048 DEBUG [hconnection-0x194ca98e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:54:05,049 DEBUG [hconnection-0x478f773c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:54:05,050 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34200, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:54:05,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:05,053 DEBUG [hconnection-0x21fce1f5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:54:05,054 DEBUG [hconnection-0x5a6855f8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:54:05,054 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34232, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:54:05,056 DEBUG [hconnection-0x5776d3f2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:54:05,056 DEBUG [hconnection-0xf7724a4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:54:05,057 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34256, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:54:05,060 DEBUG [hconnection-0x6e2ca5b6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:54:05,065 DEBUG [hconnection-0x172286ab-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:54:05,067 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34222, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:54:05,067 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34208, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:54:05,067 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34254, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:54:05,068 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34248, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:54:05,068 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34272, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:54:05,070 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34280, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:54:05,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:05,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:05,076 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:05,076 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:05,076 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:05,076 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:05,076 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:05,076 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:05,077 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:05,077 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:05,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:05,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:05,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:05,078 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:05,078 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:05,079 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:05,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:05,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:05,079 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:05,079 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:05,079 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:05,080 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:05,080 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:05,080 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:05,082 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:05,086 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:05,085 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:05,086 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. started... 2023-04-20 12:54:05,093 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4782eca030b67d2c82cb20827b0bb953 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-20 12:54:05,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 4782eca030b67d2c82cb20827b0bb953 2023-04-20 12:54:05,093 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. as already flushing 2023-04-20 12:54:05,094 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=A 2023-04-20 12:54:05,094 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:05,094 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=B 2023-04-20 12:54:05,094 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:05,094 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=C 2023-04-20 12:54:05,094 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:05,109 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681995245102, max timestamp=1681995245106Num uniques 859; 2023-04-20 12:54:05,109 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 12:54:05,109 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681995245102, max timestamp=1681995245107Num uniques 859; 2023-04-20 12:54:05,109 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 12:54:05,110 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681995245102, max timestamp=1681995245107Num uniques 859; 2023-04-20 12:54:05,110 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 12:54:05,132 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:05,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34222 deadline: 1681995305129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:05,132 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:05,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34232 deadline: 1681995305129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:05,133 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:05,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34254 deadline: 1681995305130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:05,133 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:05,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34248 deadline: 1681995305131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:05,134 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:05,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34280 deadline: 1681995305134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:05,237 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:05,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34232 deadline: 1681995305235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:05,238 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:05,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34254 deadline: 1681995305235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:05,238 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:05,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34222 deadline: 1681995305235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:05,238 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:05,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34280 deadline: 1681995305236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:05,240 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:05,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34248 deadline: 1681995305237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:05,440 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:05,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34232 deadline: 1681995305440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:05,440 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:05,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34280 deadline: 1681995305440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:05,440 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:05,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34222 deadline: 1681995305440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:05,441 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:05,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34254 deadline: 1681995305440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:05,444 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:05,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34248 deadline: 1681995305443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:05,601 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042016df5e1c35c147218d0637d743709620 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042016df5e1c35c147218d0637d743709620 2023-04-20 12:54:05,621 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=14, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/A/bdea81c3a4b5466c808ffd2f2d9707d8 2023-04-20 12:54:05,643 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/B/ddce1a4d850548148538945c50aa9cd0 2023-04-20 12:54:05,743 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:05,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34232 deadline: 1681995305741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:05,743 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:05,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34280 deadline: 1681995305741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:05,746 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:05,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34254 deadline: 1681995305744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:05,746 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:05,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34222 deadline: 1681995305744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:05,747 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:05,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34248 deadline: 1681995305747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:06,057 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/C/512e907748d84b4dbe5ad77ddc0f8db3 2023-04-20 12:54:06,065 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/A/bdea81c3a4b5466c808ffd2f2d9707d8 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/bdea81c3a4b5466c808ffd2f2d9707d8 2023-04-20 12:54:06,070 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/bdea81c3a4b5466c808ffd2f2d9707d8, entries=150, sequenceid=14, filesize=25.1 K 2023-04-20 12:54:06,072 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/B/ddce1a4d850548148538945c50aa9cd0 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/ddce1a4d850548148538945c50aa9cd0 2023-04-20 12:54:06,078 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/ddce1a4d850548148538945c50aa9cd0, entries=150, sequenceid=14, filesize=11.6 K 2023-04-20 12:54:06,079 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/C/512e907748d84b4dbe5ad77ddc0f8db3 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/512e907748d84b4dbe5ad77ddc0f8db3 2023-04-20 12:54:06,087 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/512e907748d84b4dbe5ad77ddc0f8db3, entries=150, sequenceid=14, filesize=11.6 K 2023-04-20 12:54:06,088 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 4782eca030b67d2c82cb20827b0bb953 in 1000ms, sequenceid=14, compaction requested=false 2023-04-20 12:54:06,088 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4782eca030b67d2c82cb20827b0bb953: 2023-04-20 12:54:06,088 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:06,088 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:06,088 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:06,088 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:06,089 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:06,089 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:06,090 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:06,090 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:06,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:06,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:06,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:06,091 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:06,091 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:06,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:06,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:06,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:06,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:06,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:06,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:06,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3eb51fc4[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:06,092 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:06,093 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:06,094 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:06,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:06,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:06,094 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:06,094 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:06,094 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:06,095 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:06,095 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:06,095 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:06,095 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:06,096 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:06,096 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:06,096 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:06,096 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:06,096 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:06,096 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:06,096 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:06,097 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:06,097 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:06,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:06,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:06,097 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:06,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:06,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:06,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:06,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:06,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:06,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:06,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:06,098 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:06,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:06,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:06,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:06,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:06,099 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:06,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:06,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:06,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:06,102 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:06,102 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:06,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:06,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:06,102 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:06,102 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:06,102 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:06,102 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:06,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:06,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:06,102 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:06,102 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:06,102 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:06,102 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:06,102 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:06,102 DEBUG [Thread-874] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:06,102 DEBUG [Thread-874] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:06,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:06,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:06,203 DEBUG [Thread-874] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:06,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:06,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:06,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:06,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:06,208 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:06,208 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:06,208 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:06,208 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:06,208 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:06,208 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:06,209 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:06,209 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:06,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:06,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:06,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:06,209 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:06,209 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:06,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:06,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:06,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:06,210 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:06,210 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:06,210 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:06,210 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:06,210 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:06,210 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:06,214 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:06,214 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:06,214 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:06,214 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. started... 2023-04-20 12:54:06,214 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4782eca030b67d2c82cb20827b0bb953 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-20 12:54:06,214 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=A 2023-04-20 12:54:06,214 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:54:06,214 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=B 2023-04-20 12:54:06,214 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:54:06,215 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=C 2023-04-20 12:54:06,215 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:54:06,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 4782eca030b67d2c82cb20827b0bb953 2023-04-20 12:54:06,248 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. as already flushing 2023-04-20 12:54:06,257 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:06,257 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:06,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34254 deadline: 1681995306252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:06,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34222 deadline: 1681995306252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:06,259 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:06,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34280 deadline: 1681995306257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:06,261 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:06,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34232 deadline: 1681995306257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:06,261 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:06,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34248 deadline: 1681995306258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:06,358 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:06,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34222 deadline: 1681995306358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:06,362 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:06,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34280 deadline: 1681995306360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:06,363 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:06,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34232 deadline: 1681995306362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:06,363 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:06,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34248 deadline: 1681995306362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:06,560 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:06,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34222 deadline: 1681995306560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:06,564 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-20 12:54:06,564 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:06,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34232 deadline: 1681995306564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:06,564 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:06,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34280 deadline: 1681995306564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:06,565 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:06,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34248 deadline: 1681995306565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:06,640 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420b96848c3f1ed478e9422ba592db513dc to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420b96848c3f1ed478e9422ba592db513dc 2023-04-20 12:54:06,864 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:06,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34222 deadline: 1681995306863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:06,867 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:06,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34232 deadline: 1681995306865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:06,867 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:06,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34280 deadline: 1681995306866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:06,867 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:06,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34248 deadline: 1681995306867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:07,047 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/A/ad5a56de2a324bdf9b6b49c0035c7f8d 2023-04-20 12:54:07,264 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:07,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34254 deadline: 1681995307264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:07,368 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:07,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34222 deadline: 1681995307366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:07,369 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:07,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34232 deadline: 1681995307369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:07,369 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:07,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34280 deadline: 1681995307369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:07,369 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:07,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34248 deadline: 1681995307369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:07,460 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/B/a8a61105e1b243ec9eff25b470d13888 2023-04-20 12:54:07,873 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/C/302b05d598d343c4a78fc78c11bd70e5 2023-04-20 12:54:07,878 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/A/ad5a56de2a324bdf9b6b49c0035c7f8d as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/ad5a56de2a324bdf9b6b49c0035c7f8d 2023-04-20 12:54:07,883 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/ad5a56de2a324bdf9b6b49c0035c7f8d, entries=150, sequenceid=50, filesize=25.1 K 2023-04-20 12:54:07,884 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/B/a8a61105e1b243ec9eff25b470d13888 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/a8a61105e1b243ec9eff25b470d13888 2023-04-20 12:54:07,893 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/a8a61105e1b243ec9eff25b470d13888, entries=150, sequenceid=50, filesize=11.6 K 2023-04-20 12:54:07,894 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/C/302b05d598d343c4a78fc78c11bd70e5 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/302b05d598d343c4a78fc78c11bd70e5 2023-04-20 12:54:07,899 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/302b05d598d343c4a78fc78c11bd70e5, entries=150, sequenceid=50, filesize=11.6 K 2023-04-20 12:54:07,901 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=60.38 KB/61830 for 4782eca030b67d2c82cb20827b0bb953 in 1686ms, sequenceid=50, compaction requested=false 2023-04-20 12:54:07,901 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4782eca030b67d2c82cb20827b0bb953: 2023-04-20 12:54:07,901 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:07,901 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:07,901 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:07,901 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:07,901 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:07,901 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:07,902 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:07,902 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:07,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:07,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:07,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:07,902 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:07,902 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:07,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:07,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:07,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:07,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:07,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:07,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:07,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c1bc275[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:07,903 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:07,904 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:07,904 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:07,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:07,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:07,904 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:07,904 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:07,904 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:07,904 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:07,904 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:07,905 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:07,905 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:07,905 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:07,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:07,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:07,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:07,905 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:07,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:07,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:07,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:07,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:07,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:07,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:07,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:07,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:07,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:07,909 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:07,909 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:07,909 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:07,909 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:07,909 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:07,921 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:07,921 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:07,921 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:07,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:07,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:07,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:07,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:07,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:07,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:07,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:07,922 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:07,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:07,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:07,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:07,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:07,923 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:07,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:07,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:07,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:07,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:07,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:07,935 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:07,935 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:07,935 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:07,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:07,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:07,935 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:07,935 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:07,935 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:07,935 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:07,935 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:07,935 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:07,935 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:07,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:07,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:07,936 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:07,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:07,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:07,937 DEBUG [Thread-874] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:07,937 DEBUG [Thread-874] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:08,038 DEBUG [Thread-874] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:08,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:08,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:08,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:08,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:08,043 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:08,043 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:08,043 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:08,043 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:08,043 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,043 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:08,044 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:08,044 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:08,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:08,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:08,044 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,045 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:08,045 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:08,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:08,045 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:08,047 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:08,047 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:08,047 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:08,047 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:08,047 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:08,048 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:08,048 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:08,048 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:08,048 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. started... 2023-04-20 12:54:08,048 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4782eca030b67d2c82cb20827b0bb953 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-20 12:54:08,048 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=A 2023-04-20 12:54:08,048 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:08,048 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=B 2023-04-20 12:54:08,049 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:08,049 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=C 2023-04-20 12:54:08,049 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:08,092 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420fcc1275c57894401ac37098bf162bea4 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420fcc1275c57894401ac37098bf162bea4 2023-04-20 12:54:08,104 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=62, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/A/0ee6e07d36254035b35e1d2f027596c8 2023-04-20 12:54:08,123 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/B/304fd72e2ee442599ff22e660d7efaa8 2023-04-20 12:54:08,150 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/C/99970ffb2c2d4b059afff7e67052475f 2023-04-20 12:54:08,158 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/A/0ee6e07d36254035b35e1d2f027596c8 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/0ee6e07d36254035b35e1d2f027596c8 2023-04-20 12:54:08,166 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/0ee6e07d36254035b35e1d2f027596c8, entries=150, sequenceid=62, filesize=25.1 K 2023-04-20 12:54:08,168 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/B/304fd72e2ee442599ff22e660d7efaa8 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/304fd72e2ee442599ff22e660d7efaa8 2023-04-20 12:54:08,177 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/304fd72e2ee442599ff22e660d7efaa8, entries=150, sequenceid=62, filesize=11.6 K 2023-04-20 12:54:08,179 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/C/99970ffb2c2d4b059afff7e67052475f as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/99970ffb2c2d4b059afff7e67052475f 2023-04-20 12:54:08,185 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/99970ffb2c2d4b059afff7e67052475f, entries=150, sequenceid=62, filesize=11.6 K 2023-04-20 12:54:08,186 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for 4782eca030b67d2c82cb20827b0bb953 in 138ms, sequenceid=62, compaction requested=true 2023-04-20 12:54:08,186 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4782eca030b67d2c82cb20827b0bb953: 2023-04-20 12:54:08,186 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:08,187 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:08,187 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:08,187 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,187 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:08,187 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:08,188 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,188 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:08,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:08,188 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,188 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:08,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:08,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:08,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:08,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:08,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:08,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@599a61d7[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:08,190 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:08,190 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,191 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,191 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:08,191 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:08,191 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:08,191 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,191 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:08,192 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,192 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:08,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:08,192 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:08,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:08,192 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:08,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:08,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:08,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:08,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:08,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:08,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:08,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:08,195 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:08,195 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:08,195 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:08,195 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:08,195 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:08,196 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,196 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,197 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:08,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:08,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:08,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:08,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:08,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:08,197 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:08,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:08,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:08,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,199 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:08,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:08,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,201 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:08,201 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:08,201 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,201 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:08,201 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:08,201 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:08,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:08,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:08,201 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:08,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:08,201 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,202 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,202 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,202 DEBUG [Thread-874] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:08,202 DEBUG [Thread-874] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:08,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:08,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:08,302 DEBUG [Thread-874] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:08,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:08,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:08,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:08,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:08,306 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:08,306 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:08,307 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:08,307 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:08,307 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,307 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:08,308 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:08,308 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:08,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:08,308 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,308 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:08,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:08,309 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:08,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:08,309 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:08,310 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:08,310 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:08,310 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:08,310 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:08,310 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:08,318 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:08,318 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:08,318 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:08,318 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. started... 2023-04-20 12:54:08,318 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4782eca030b67d2c82cb20827b0bb953: 2023-04-20 12:54:08,318 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:08,318 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:08,318 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:08,318 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,318 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:08,318 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:08,326 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,326 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:08,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:08,326 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,326 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:08,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:08,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:08,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:08,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:08,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:08,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62272f96[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:08,328 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:08,328 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,329 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,329 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:08,329 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:08,329 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,329 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:08,329 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:08,330 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,330 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:08,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:08,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:08,330 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:08,331 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:08,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:08,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:08,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:08,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:08,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:08,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:08,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:08,333 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:08,333 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:08,333 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:08,333 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:08,333 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:08,334 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,334 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:08,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,334 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:08,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:08,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:08,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:08,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:08,335 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:08,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:08,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:08,337 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:08,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:08,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,346 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,346 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:08,346 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:08,346 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,346 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,346 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,346 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:08,346 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:08,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:08,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:08,346 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:08,346 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:08,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:08,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:08,346 DEBUG [Thread-874] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:08,347 DEBUG [Thread-874] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:08,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:08,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:08,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 4782eca030b67d2c82cb20827b0bb953 2023-04-20 12:54:08,413 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4782eca030b67d2c82cb20827b0bb953 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-20 12:54:08,414 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=A 2023-04-20 12:54:08,415 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:08,415 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=B 2023-04-20 12:54:08,415 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:08,415 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=C 2023-04-20 12:54:08,415 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:08,447 DEBUG [Thread-874] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:08,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:08,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:08,451 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:08,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34280 deadline: 1681995308445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,451 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:08,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34248 deadline: 1681995308445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:08,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681995248438, max timestamp=1681995248449Num uniques 859; 2023-04-20 12:54:08,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:08,452 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:08,452 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:08,452 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:08,452 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:08,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 12:54:08,452 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,452 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:08,452 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304201ae1f2e7a5564484ab1a9e0f48ab07bf to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304201ae1f2e7a5564484ab1a9e0f48ab07bf 2023-04-20 12:54:08,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681995248438, max timestamp=1681995248449Num uniques 859; 2023-04-20 12:54:08,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 12:54:08,453 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:08,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681995248438, max timestamp=1681995248449Num uniques 859; 2023-04-20 12:54:08,453 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:08,453 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 12:54:08,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:08,454 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,454 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:08,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:08,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:08,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:08,456 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:08,459 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:08,459 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:08,459 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:08,459 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:08,459 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:08,462 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:08,462 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:08,462 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:08,462 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. started... 2023-04-20 12:54:08,462 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. as already flushing 2023-04-20 12:54:08,462 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:08,462 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:08,462 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:08,462 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,462 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:08,462 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:08,467 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:08,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:08,467 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:08,468 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,468 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:08,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:08,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:08,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:08,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:08,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74fa5729[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:08,469 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:08,469 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,469 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,470 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:08,470 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:08,470 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,470 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:08,470 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:08,470 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,470 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:08,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:08,470 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:08,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:08,471 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:08,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:08,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:08,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:08,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:08,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:08,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:08,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:08,472 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:08,473 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:08,473 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:08,473 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:08,473 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:08,474 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,474 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:08,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,474 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:08,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:08,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:08,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:08,474 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:08,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:08,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:08,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:08,475 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:08,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:08,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,477 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,477 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:08,477 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:08,477 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:08,477 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:08,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:08,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:08,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:08,477 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:08,477 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,477 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,477 DEBUG [Thread-874] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:08,477 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,477 DEBUG [Thread-874] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:08,478 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:08,478 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:08,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:08,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:08,483 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:08,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34232 deadline: 1681995308478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,484 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:08,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34222 deadline: 1681995308479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,553 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:08,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34280 deadline: 1681995308552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,553 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:08,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34248 deadline: 1681995308552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,578 DEBUG [Thread-874] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:08,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:08,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:08,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:08,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:08,581 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:08,581 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:08,581 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:08,582 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:08,583 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,583 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:08,584 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:08,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:08,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:08,584 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:08,584 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,585 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:08,585 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:08,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:08,585 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:08,585 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:08,585 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:08,585 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:08,585 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:08,585 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:08,588 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:08,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34232 deadline: 1681995308584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,590 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:08,590 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:08,590 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:08,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34222 deadline: 1681995308585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,590 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:08,590 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. started... 2023-04-20 12:54:08,590 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. as already flushing 2023-04-20 12:54:08,590 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:08,590 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:08,590 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:08,590 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,590 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:08,590 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:08,592 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,592 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:08,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:08,592 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,592 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:08,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:08,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:08,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:08,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:08,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:08,594 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:08,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56502ba9[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:08,594 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,594 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,594 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:08,594 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:08,594 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:08,594 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,595 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:08,595 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,595 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:08,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,595 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:08,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:08,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:08,595 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:08,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:08,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:08,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:08,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:08,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:08,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:08,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:08,597 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:08,597 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:08,597 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:08,597 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:08,597 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:08,598 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,598 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:08,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:08,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,598 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:08,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:08,599 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:08,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:08,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:08,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:08,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:08,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,600 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:08,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:08,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,602 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,602 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:08,602 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:08,602 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,602 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,602 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:08,602 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:08,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:08,602 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,602 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:08,603 DEBUG [Thread-874] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:08,603 DEBUG [Thread-874] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:08,603 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:08,603 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:08,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:08,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:08,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:08,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:08,703 DEBUG [Thread-874] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:08,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:08,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:08,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:08,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:08,706 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:08,706 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:08,706 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:08,706 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:08,707 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,707 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:08,707 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:08,707 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:08,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:08,708 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,708 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:08,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:08,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:08,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:08,708 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:08,709 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:08,709 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:08,709 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:08,709 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:08,709 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:08,709 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:08,709 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:08,709 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:08,709 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. started... 2023-04-20 12:54:08,709 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. as already flushing 2023-04-20 12:54:08,709 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:08,710 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:08,710 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:08,710 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,710 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:08,710 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:08,711 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,711 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:08,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:08,711 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,711 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:08,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:08,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:08,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:08,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:08,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:08,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c4e6c6d[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:08,713 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:08,713 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,713 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,713 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:08,713 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:08,714 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:08,714 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,714 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:08,714 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:08,714 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,714 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:08,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:08,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:08,715 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:08,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:08,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:08,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:08,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:08,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:08,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:08,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:08,716 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:08,716 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:08,717 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:08,717 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:08,717 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:08,717 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,718 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:08,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,718 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:08,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:08,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:08,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:08,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:08,718 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:08,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:08,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:08,720 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:08,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:08,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,722 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:08,722 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:08,722 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,722 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:08,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:08,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:08,722 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:08,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:08,722 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:08,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:08,722 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,723 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,723 DEBUG [Thread-874] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:08,723 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,723 DEBUG [Thread-874] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:08,723 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:08,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:08,756 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:08,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34280 deadline: 1681995308755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,757 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:08,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34248 deadline: 1681995308755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:08,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34232 deadline: 1681995308790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,792 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:08,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34222 deadline: 1681995308791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,823 DEBUG [Thread-874] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:08,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:08,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:08,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:08,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:08,826 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:08,826 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:08,826 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:08,826 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:08,827 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,827 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:08,827 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,827 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:08,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:08,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:08,828 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,828 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:08,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:08,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:08,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:08,828 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:08,829 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:08,829 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:08,829 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:08,829 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:08,829 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:08,829 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:08,829 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:08,829 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:08,829 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. started... 2023-04-20 12:54:08,829 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. as already flushing 2023-04-20 12:54:08,829 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:08,829 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:08,829 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:08,829 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,829 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:08,829 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:08,830 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,830 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:08,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:08,830 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,830 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:08,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:08,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:08,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:08,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:08,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:08,832 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:08,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74eb3ed[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:08,832 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,833 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,833 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:08,833 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:08,833 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:08,833 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,833 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:08,834 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,834 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:08,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:08,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:08,834 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:08,834 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:08,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:08,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:08,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:08,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:08,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:08,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:08,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:08,835 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:08,835 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:08,836 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:08,836 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:08,836 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:08,836 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,836 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:08,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:08,836 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:08,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:08,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:08,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:08,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,837 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:08,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:08,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:08,839 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:08,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:08,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,840 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,840 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:08,840 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:08,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:08,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:08,840 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:08,840 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:08,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:08,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:08,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:08,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:08,841 DEBUG [Thread-874] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:08,841 DEBUG [Thread-874] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:08,840 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:08,840 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:08,842 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,842 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,842 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,842 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,865 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=74, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/A/4699f61e230c48a3af60b3d01f5a6187 2023-04-20 12:54:08,941 DEBUG [Thread-874] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:08,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:08,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:08,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:08,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:08,944 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:08,944 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:08,944 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:08,944 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:08,958 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,958 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:08,959 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:08,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:08,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:08,959 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:08,960 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,960 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:08,960 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:08,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:08,960 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:08,960 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:08,960 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:08,960 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:08,961 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:08,961 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:08,961 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:08,961 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:08,961 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:08,961 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. started... 2023-04-20 12:54:08,961 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. as already flushing 2023-04-20 12:54:08,961 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:08,961 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:08,961 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:08,961 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,961 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:08,961 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:08,962 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,962 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:08,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:08,962 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,962 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:08,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:08,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:08,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:08,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:08,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:08,963 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:08,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@212bff32[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:08,963 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,965 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,965 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:08,965 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,965 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:08,965 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:08,965 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:08,966 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:08,966 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:08,966 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,966 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:08,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:08,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:08,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:08,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:08,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:08,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:08,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:08,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:08,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:08,970 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:08,970 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:08,970 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:08,970 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:08,970 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:08,976 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,976 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:08,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:08,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,976 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:08,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:08,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:08,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:08,977 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:08,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:08,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:08,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:08,978 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:08,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:08,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,980 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:08,980 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:08,980 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:08,980 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:08,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:08,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:08,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:08,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:08,980 DEBUG [Thread-874] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:08,980 DEBUG [Thread-874] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:08,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:08,980 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:08,980 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:08,980 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:08,981 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:08,981 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,981 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:08,981 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:09,060 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:09,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34280 deadline: 1681995309057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:09,061 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:09,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34248 deadline: 1681995309059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:09,080 DEBUG [Thread-874] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:09,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:09,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:09,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:09,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:09,084 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:09,084 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:09,084 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:09,084 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:09,086 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:09,086 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:09,086 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:09,086 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:09,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:09,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:09,087 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:09,087 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:09,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:09,087 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:09,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:09,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:09,087 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:09,088 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:09,088 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:09,088 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:09,088 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:09,088 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:09,088 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:09,088 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:09,088 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:09,088 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. started... 2023-04-20 12:54:09,088 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. as already flushing 2023-04-20 12:54:09,088 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:09,088 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:09,088 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:09,088 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:09,088 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:09,088 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:09,089 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:09,089 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:09,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:09,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:09,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:09,090 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:09,090 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:09,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:09,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:09,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:09,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:09,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:09,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:09,091 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:09,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76f185dd[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:09,091 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:09,092 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:09,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:09,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:09,092 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:09,092 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:09,092 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:09,092 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:09,092 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:09,093 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:09,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:09,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:09,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:09,093 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:09,093 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:09,093 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:09,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:09,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:09,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:09,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:09,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:09,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:09,095 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:09,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:09,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34222 deadline: 1681995309093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:09,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:09,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:09,095 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:09,095 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:09,095 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:09,095 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:09,095 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:09,096 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:09,096 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:09,096 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:09,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34232 deadline: 1681995309094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:09,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:09,097 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:09,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:09,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:09,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:09,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:09,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:09,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:09,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:09,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:09,098 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:09,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:09,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:09,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:09,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:09,098 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:09,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:09,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:09,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:09,100 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:09,100 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:09,100 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:09,100 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:09,100 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:09,100 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:09,100 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:09,100 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:09,100 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:09,100 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:09,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:09,100 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:09,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:09,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:09,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:09,101 DEBUG [Thread-874] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:09,101 DEBUG [Thread-874] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:09,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:09,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:09,201 DEBUG [Thread-874] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:09,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:09,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:09,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:09,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:09,205 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:09,205 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:09,205 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:09,205 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:09,205 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:09,206 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:09,206 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:09,206 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:09,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:09,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:09,206 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:09,206 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:09,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:09,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:09,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:09,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:09,207 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:09,207 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:09,207 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:09,208 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:09,208 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:09,208 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:09,208 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:09,208 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:09,208 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:09,208 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. started... 2023-04-20 12:54:09,208 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. as already flushing 2023-04-20 12:54:09,208 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:09,208 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:09,208 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:09,208 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:09,208 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:09,208 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:09,209 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:09,209 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:09,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:09,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:09,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:09,209 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:09,209 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:09,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:09,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:09,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:09,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:09,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:09,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:09,210 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:09,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5973a26a[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:09,210 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:09,211 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:09,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:09,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:09,211 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:09,211 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:09,211 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:09,211 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:09,211 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:09,212 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:09,212 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:09,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:09,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:09,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:09,212 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:09,212 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:09,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:09,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:09,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:09,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:09,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:09,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:09,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:09,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:09,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:09,213 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:09,214 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:09,214 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:09,214 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:09,214 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:09,214 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:09,214 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:09,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:09,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:09,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:09,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:09,214 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:09,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:09,215 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:09,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:09,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:09,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:09,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:09,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:09,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:09,216 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:09,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:09,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:09,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:09,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:09,217 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:09,217 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:09,218 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:09,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:09,218 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:09,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:09,224 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:09,224 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:09,224 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:09,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:09,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:09,224 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:09,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:09,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:09,224 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:09,224 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:09,224 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:09,225 DEBUG [Thread-874] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:09,225 DEBUG [Thread-874] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:09,276 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=74 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/B/56bf275d835a4bca911d458559450d17 2023-04-20 12:54:09,277 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:09,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34254 deadline: 1681995309275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:09,278 DEBUG [Thread-866] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4147 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953., hostname=jenkins-hbase20.apache.org,36277,1681995213947, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-20 12:54:09,287 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=74 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/C/41035605d1784bc38301d3ab4a68d6f3 2023-04-20 12:54:09,298 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/A/4699f61e230c48a3af60b3d01f5a6187 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/4699f61e230c48a3af60b3d01f5a6187 2023-04-20 12:54:09,302 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/4699f61e230c48a3af60b3d01f5a6187, entries=400, sequenceid=74, filesize=59.0 K 2023-04-20 12:54:09,303 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/B/56bf275d835a4bca911d458559450d17 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/56bf275d835a4bca911d458559450d17 2023-04-20 12:54:09,307 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/56bf275d835a4bca911d458559450d17, entries=150, sequenceid=74, filesize=11.6 K 2023-04-20 12:54:09,308 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/C/41035605d1784bc38301d3ab4a68d6f3 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/41035605d1784bc38301d3ab4a68d6f3 2023-04-20 12:54:09,312 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/41035605d1784bc38301d3ab4a68d6f3, entries=150, sequenceid=74, filesize=11.6 K 2023-04-20 12:54:09,313 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 4782eca030b67d2c82cb20827b0bb953 in 900ms, sequenceid=74, compaction requested=true 2023-04-20 12:54:09,313 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4782eca030b67d2c82cb20827b0bb953: 2023-04-20 12:54:09,313 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:09,313 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 12:54:09,313 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 12:54:09,313 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:09,313 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:54:09,314 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 12:54:09,314 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): 4782eca030b67d2c82cb20827b0bb953/B is initiating minor compaction (all files) 2023-04-20 12:54:09,314 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4782eca030b67d2c82cb20827b0bb953/B in TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:09,314 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/ddce1a4d850548148538945c50aa9cd0, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/a8a61105e1b243ec9eff25b470d13888, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/304fd72e2ee442599ff22e660d7efaa8, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/56bf275d835a4bca911d458559450d17] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp, totalSize=46.4 K 2023-04-20 12:54:09,315 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 137692 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 12:54:09,315 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): 4782eca030b67d2c82cb20827b0bb953/A is initiating minor compaction (all files) 2023-04-20 12:54:09,315 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4782eca030b67d2c82cb20827b0bb953/A in TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:09,315 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/bdea81c3a4b5466c808ffd2f2d9707d8, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/ad5a56de2a324bdf9b6b49c0035c7f8d, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/0ee6e07d36254035b35e1d2f027596c8, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/4699f61e230c48a3af60b3d01f5a6187] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp, totalSize=134.5 K 2023-04-20 12:54:09,315 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting ddce1a4d850548148538945c50aa9cd0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1681995245090 2023-04-20 12:54:09,315 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting bdea81c3a4b5466c808ffd2f2d9707d8, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1681995245090 2023-04-20 12:54:09,315 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting a8a61105e1b243ec9eff25b470d13888, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681995245128 2023-04-20 12:54:09,316 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting ad5a56de2a324bdf9b6b49c0035c7f8d, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681995245128 2023-04-20 12:54:09,316 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 304fd72e2ee442599ff22e660d7efaa8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681995246255 2023-04-20 12:54:09,316 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 0ee6e07d36254035b35e1d2f027596c8, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681995246255 2023-04-20 12:54:09,316 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 56bf275d835a4bca911d458559450d17, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=74, earliestPutTs=1681995248404 2023-04-20 12:54:09,316 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 4699f61e230c48a3af60b3d01f5a6187, keycount=400, bloomtype=ROW, size=59.0 K, encoding=NONE, compression=NONE, seqNum=74, earliestPutTs=1681995248400 2023-04-20 12:54:09,325 DEBUG [Thread-874] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:09,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:09,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:09,341 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): 4782eca030b67d2c82cb20827b0bb953#B#compaction#119 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:09,345 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4782eca030b67d2c82cb20827b0bb953#A#compaction#118 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:09,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:09,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:09,345 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:09,345 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:09,345 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:09,345 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:09,346 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:09,346 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:09,347 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:09,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:09,347 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:09,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:09,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:09,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:09,348 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:09,348 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:09,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:09,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:09,348 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:09,359 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:09,359 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:09,359 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:09,359 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:09,359 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:09,362 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:09,362 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:09,362 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:09,362 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. started... 2023-04-20 12:54:09,363 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4782eca030b67d2c82cb20827b0bb953 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-20 12:54:09,363 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=A 2023-04-20 12:54:09,363 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:54:09,363 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=B 2023-04-20 12:54:09,363 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:54:09,363 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=C 2023-04-20 12:54:09,364 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:54:09,371 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/B/7c9780a9f8f842c09b2e5b444f6f33d6 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/7c9780a9f8f842c09b2e5b444f6f33d6 2023-04-20 12:54:09,377 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4782eca030b67d2c82cb20827b0bb953/B of 4782eca030b67d2c82cb20827b0bb953 into 7c9780a9f8f842c09b2e5b444f6f33d6(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:54:09,377 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4782eca030b67d2c82cb20827b0bb953: 2023-04-20 12:54:09,377 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953., storeName=4782eca030b67d2c82cb20827b0bb953/B, priority=12, startTime=1681995249313; duration=0sec 2023-04-20 12:54:09,378 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:54:09,378 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 12:54:09,379 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 12:54:09,379 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): 4782eca030b67d2c82cb20827b0bb953/C is initiating minor compaction (all files) 2023-04-20 12:54:09,379 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4782eca030b67d2c82cb20827b0bb953/C in TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:09,379 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/512e907748d84b4dbe5ad77ddc0f8db3, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/302b05d598d343c4a78fc78c11bd70e5, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/99970ffb2c2d4b059afff7e67052475f, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/41035605d1784bc38301d3ab4a68d6f3] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp, totalSize=46.4 K 2023-04-20 12:54:09,380 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 512e907748d84b4dbe5ad77ddc0f8db3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1681995245090 2023-04-20 12:54:09,380 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 302b05d598d343c4a78fc78c11bd70e5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681995245128 2023-04-20 12:54:09,380 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 99970ffb2c2d4b059afff7e67052475f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1681995246255 2023-04-20 12:54:09,381 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 41035605d1784bc38301d3ab4a68d6f3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=74, earliestPutTs=1681995248404 2023-04-20 12:54:09,400 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): 4782eca030b67d2c82cb20827b0bb953#C#compaction#121 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:09,414 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/A/ee0511af6fa54322a01dd6f4429cf8c3 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/ee0511af6fa54322a01dd6f4429cf8c3 2023-04-20 12:54:09,420 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4782eca030b67d2c82cb20827b0bb953/A of 4782eca030b67d2c82cb20827b0bb953 into ee0511af6fa54322a01dd6f4429cf8c3(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:54:09,420 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4782eca030b67d2c82cb20827b0bb953: 2023-04-20 12:54:09,420 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953., storeName=4782eca030b67d2c82cb20827b0bb953/A, priority=12, startTime=1681995249313; duration=0sec 2023-04-20 12:54:09,421 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:09,436 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/C/4c5eb47fd0fc45b592c4095368ef7946 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/4c5eb47fd0fc45b592c4095368ef7946 2023-04-20 12:54:09,446 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 4782eca030b67d2c82cb20827b0bb953/C of 4782eca030b67d2c82cb20827b0bb953 into 4c5eb47fd0fc45b592c4095368ef7946(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:54:09,446 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4782eca030b67d2c82cb20827b0bb953: 2023-04-20 12:54:09,446 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953., storeName=4782eca030b67d2c82cb20827b0bb953/C, priority=12, startTime=1681995249313; duration=0sec 2023-04-20 12:54:09,446 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:09,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 4782eca030b67d2c82cb20827b0bb953 2023-04-20 12:54:09,568 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. as already flushing 2023-04-20 12:54:09,575 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:09,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34280 deadline: 1681995309575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:09,577 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:09,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34248 deadline: 1681995309575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:09,598 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:09,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34222 deadline: 1681995309598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:09,602 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:09,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34232 deadline: 1681995309600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:09,690 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:09,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34248 deadline: 1681995309689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:09,690 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:09,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34280 deadline: 1681995309690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:09,800 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420073c5bf151ba4ab0a386a6ac0b6c39cd to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420073c5bf151ba4ab0a386a6ac0b6c39cd 2023-04-20 12:54:09,891 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:09,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34248 deadline: 1681995309891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:09,892 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:09,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34280 deadline: 1681995309891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,195 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:10,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34248 deadline: 1681995310194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,198 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:10,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34280 deadline: 1681995310195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,205 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=110, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/A/80a7470001324a8d9fb0a7d41e86a431 2023-04-20 12:54:10,602 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:10,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34222 deadline: 1681995310601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,604 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:10,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34232 deadline: 1681995310603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,615 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=110 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/B/1368f72210ee45e5bdbace57728fc31d 2023-04-20 12:54:10,637 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=110 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/C/bb39d0be8dfe41ae9cebe9cdd7286a09 2023-04-20 12:54:10,641 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/A/80a7470001324a8d9fb0a7d41e86a431 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/80a7470001324a8d9fb0a7d41e86a431 2023-04-20 12:54:10,646 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/80a7470001324a8d9fb0a7d41e86a431, entries=150, sequenceid=110, filesize=25.1 K 2023-04-20 12:54:10,647 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/B/1368f72210ee45e5bdbace57728fc31d as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/1368f72210ee45e5bdbace57728fc31d 2023-04-20 12:54:10,651 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/1368f72210ee45e5bdbace57728fc31d, entries=150, sequenceid=110, filesize=11.6 K 2023-04-20 12:54:10,652 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/C/bb39d0be8dfe41ae9cebe9cdd7286a09 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/bb39d0be8dfe41ae9cebe9cdd7286a09 2023-04-20 12:54:10,656 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/bb39d0be8dfe41ae9cebe9cdd7286a09, entries=150, sequenceid=110, filesize=11.6 K 2023-04-20 12:54:10,657 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 4782eca030b67d2c82cb20827b0bb953 in 1294ms, sequenceid=110, compaction requested=false 2023-04-20 12:54:10,657 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4782eca030b67d2c82cb20827b0bb953: 2023-04-20 12:54:10,657 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:10,657 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:10,658 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:10,658 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,661 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:10,661 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:10,662 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,662 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:10,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:10,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:10,663 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:10,663 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:10,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:10,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:10,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:10,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:10,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:10,664 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:10,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b50b372[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:10,664 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:10,665 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:10,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:10,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:10,665 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:10,665 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:10,665 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:10,665 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,665 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:10,666 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,666 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:10,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:10,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:10,667 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:10,667 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:10,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:10,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:10,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:10,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:10,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:10,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:10,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:10,669 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:10,669 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:10,669 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:10,669 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:10,669 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:10,670 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:10,670 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:10,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:10,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:10,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:10,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:10,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:10,670 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:10,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:10,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:10,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:10,670 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:10,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:10,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:10,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:10,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,671 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:10,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:10,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,673 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:10,673 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:10,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:10,673 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:10,673 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:10,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:10,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:10,673 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:10,673 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:10,673 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,673 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:10,673 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:10,673 DEBUG [Thread-874] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:10,673 DEBUG [Thread-874] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:10,673 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:10,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:10,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:10,674 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:10,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 4782eca030b67d2c82cb20827b0bb953 2023-04-20 12:54:10,699 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4782eca030b67d2c82cb20827b0bb953 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-20 12:54:10,699 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=A 2023-04-20 12:54:10,699 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:10,699 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=B 2023-04-20 12:54:10,699 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:10,699 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=C 2023-04-20 12:54:10,699 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:10,737 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:10,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34280 deadline: 1681995310735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,737 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:10,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34248 deadline: 1681995310736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,773 DEBUG [Thread-874] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:10,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:10,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:10,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:10,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:10,777 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:10,777 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:10,777 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:10,777 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:10,777 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:10,777 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:10,778 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,778 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:10,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:10,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:10,778 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,778 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:10,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:10,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:10,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:10,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:10,779 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:10,779 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:10,779 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:10,779 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:10,779 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:10,779 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:10,779 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:10,779 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:10,779 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:10,779 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. started... 2023-04-20 12:54:10,779 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. as already flushing 2023-04-20 12:54:10,779 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:10,779 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:10,780 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:10,780 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,780 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:10,780 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:10,780 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,780 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:10,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:10,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:10,781 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:10,781 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:10,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:10,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:10,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:10,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:10,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:10,782 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:10,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@557c0b45[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:10,782 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:10,782 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:10,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:10,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:10,782 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:10,782 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:10,783 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:10,783 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,783 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:10,784 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,784 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:10,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:10,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:10,784 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:10,784 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:10,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:10,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:10,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:10,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:10,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:10,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:10,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:10,785 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:10,786 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:10,786 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:10,786 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:10,786 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:10,787 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:10,787 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:10,787 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:10,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:10,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:10,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:10,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:10,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:10,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:10,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:10,787 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:10,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:10,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:10,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:10,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:10,794 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:10,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:10,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,800 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,800 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:10,800 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:10,800 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,800 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:10,801 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:10,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:10,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:10,800 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:10,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:10,801 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:10,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:10,801 DEBUG [Thread-874] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:10,801 DEBUG [Thread-874] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:10,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:10,801 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:10,801 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:10,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:10,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:10,839 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:10,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34280 deadline: 1681995310838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,840 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:10,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34248 deadline: 1681995310838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,901 DEBUG [Thread-874] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:10,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:10,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:10,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:10,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:10,904 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:10,904 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:10,904 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:10,904 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:10,904 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:10,904 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:10,905 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:10,905 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:10,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:10,905 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,905 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:10,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:10,906 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:10,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:10,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:10,906 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:10,906 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:10,906 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:10,906 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:10,906 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:10,906 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:10,906 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:10,906 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:10,906 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:10,906 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. started... 2023-04-20 12:54:10,906 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. as already flushing 2023-04-20 12:54:10,907 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:10,907 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:10,907 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:10,907 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,907 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:10,907 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:10,908 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,908 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:10,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:10,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:10,908 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:10,908 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:10,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:10,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:10,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:10,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:10,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:10,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f59a35a[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:10,909 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:10,909 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:10,910 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:10,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:10,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:10,910 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:10,910 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:10,910 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:10,910 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,910 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:10,911 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,911 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:10,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:10,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:10,911 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:10,911 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:10,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:10,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:10,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:10,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:10,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:10,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:10,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:10,913 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:10,913 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:10,913 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:10,913 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:10,913 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:10,913 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:10,913 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:10,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:10,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:10,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:10,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:10,913 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:10,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:10,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:10,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:10,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:10,914 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:10,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:10,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:10,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:10,915 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:10,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:10,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,916 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,916 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:10,916 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:10,916 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:10,916 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:10,916 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:10,916 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:10,916 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:10,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:10,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:10,917 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:10,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:10,917 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:10,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:10,917 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:10,917 DEBUG [Thread-874] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:10,917 DEBUG [Thread-874] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:10,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:10,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:11,017 DEBUG [Thread-874] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:11,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:11,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:11,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:11,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:11,020 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:11,020 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:11,020 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:11,020 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:11,021 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:11,021 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:11,021 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:11,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:11,021 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:11,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:11,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:11,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:11,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:11,023 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:11,023 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,023 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:11,023 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:11,023 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:11,023 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:11,023 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:11,023 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:11,023 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:11,023 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:11,023 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:11,024 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. started... 2023-04-20 12:54:11,024 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. as already flushing 2023-04-20 12:54:11,024 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:11,024 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:11,024 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:11,024 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,024 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:11,024 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:11,025 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:11,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:11,025 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:11,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:11,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:11,026 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:11,026 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:11,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:11,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:11,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:11,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b7d3429[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:11,027 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:11,027 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:11,028 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:11,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:11,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:11,028 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,028 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:11,028 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:11,028 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:11,028 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:11,029 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:11,029 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,029 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:11,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:11,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:11,029 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:11,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:11,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:11,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:11,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:11,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:11,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:11,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:11,032 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:11,032 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:11,032 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:11,032 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:11,033 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:11,034 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:11,034 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:11,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:11,034 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:11,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:11,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:11,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:11,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:11,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:11,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:11,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:11,035 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,036 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:11,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:11,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:11,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:11,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,039 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,039 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:11,039 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:11,039 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,039 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:11,039 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:11,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:11,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:11,039 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:11,039 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:11,039 DEBUG [Thread-874] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:11,039 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:11,039 DEBUG [Thread-874] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:11,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:11,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:11,039 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:11,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:11,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:11,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:11,040 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:11,043 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:11,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34280 deadline: 1681995311041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,045 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:11,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34248 deadline: 1681995311042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,111 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420477cdddc064f4877a982ffe62083577e to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420477cdddc064f4877a982ffe62083577e 2023-04-20 12:54:11,139 DEBUG [Thread-874] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:11,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:11,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:11,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:11,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:11,142 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:11,142 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:11,143 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:11,143 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:11,143 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:11,143 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:11,144 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:11,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:11,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:11,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:11,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:11,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:11,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:11,145 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:11,146 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:11,146 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:11,146 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:11,146 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:11,146 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:11,146 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:11,146 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:11,146 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:11,146 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. started... 2023-04-20 12:54:11,147 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. as already flushing 2023-04-20 12:54:11,147 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:11,147 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:11,147 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:11,147 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,147 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:11,147 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:11,148 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,148 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:11,149 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:11,149 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:11,149 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,149 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:11,149 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b594143[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:11,149 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:11,149 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:11,149 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:11,149 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:11,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:11,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:11,150 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:11,150 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:11,150 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,150 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:11,150 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:11,153 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,153 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:11,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:11,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:11,153 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:11,153 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:11,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:11,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:11,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:11,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:11,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:11,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:11,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:11,155 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:11,156 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:11,156 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:11,156 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:11,156 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:11,156 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:11,156 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:11,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:11,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:11,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:11,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:11,156 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:11,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:11,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:11,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:11,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:11,157 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:11,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:11,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:11,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:11,158 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:11,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:11,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,159 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:11,159 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:11,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:11,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:11,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:11,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:11,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:11,159 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,159 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:11,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:11,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:11,160 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:11,160 DEBUG [Thread-874] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:11,160 DEBUG [Thread-874] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:11,160 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:11,160 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,160 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:11,160 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:11,160 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:11,260 DEBUG [Thread-874] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:11,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:11,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:11,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:11,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:11,263 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:11,263 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:11,263 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:11,263 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:11,264 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:11,264 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:11,264 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:11,264 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:11,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:11,265 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:11,265 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:11,265 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:11,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:11,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:11,265 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:11,265 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:11,265 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:11,265 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:11,265 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:11,266 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:11,266 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:11,266 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:11,266 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:11,266 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. started... 2023-04-20 12:54:11,266 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. as already flushing 2023-04-20 12:54:11,266 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:11,266 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:11,266 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:11,266 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,266 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:11,266 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:11,267 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:11,267 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:11,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:11,267 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:11,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:11,267 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:11,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:11,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:11,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:11,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:11,268 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:11,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4518d117[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:11,268 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:11,269 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:11,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:11,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:11,269 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:11,269 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,269 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:11,269 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:11,269 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:11,269 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,269 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:11,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:11,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:11,270 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:11,270 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:11,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:11,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:11,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:11,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:11,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:11,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:11,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:11,279 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:11,279 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:11,279 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:11,279 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:11,279 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:11,296 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:11,296 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:11,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:11,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:11,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:11,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:11,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:11,296 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:11,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:11,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:11,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:11,297 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:11,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:11,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:11,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:11,298 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:11,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:11,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,304 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,304 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:11,304 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:11,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:11,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:11,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:11,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:11,304 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:11,304 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:11,305 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:11,305 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,305 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:11,305 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:11,305 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:11,305 DEBUG [Thread-874] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:11,305 DEBUG [Thread-874] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:11,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:11,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:11,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:11,345 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:11,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34280 deadline: 1681995311344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,351 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:11,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34248 deadline: 1681995311348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,405 DEBUG [Thread-874] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:11,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:11,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:11,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:11,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:11,408 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:11,408 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:11,408 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:11,408 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:11,409 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:11,409 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:11,409 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:11,409 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:11,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:11,410 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,410 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:11,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:11,410 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:11,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:11,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:11,410 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:11,411 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:11,411 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:11,411 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:11,411 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:11,411 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:11,412 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:11,412 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:11,412 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:11,412 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. started... 2023-04-20 12:54:11,412 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. as already flushing 2023-04-20 12:54:11,412 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:11,412 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:11,412 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:11,412 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,412 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:11,412 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:11,414 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,414 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:11,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:11,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:11,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:11,415 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:11,415 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:11,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:11,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:11,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:11,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:11,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10f440fa[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:11,417 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:11,417 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:11,417 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:11,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:11,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:11,417 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:11,418 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:11,418 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:11,418 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,418 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:11,419 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,419 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:11,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:11,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:11,419 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:11,419 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:11,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:11,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:11,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:11,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:11,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:11,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:11,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:11,421 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:11,421 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:11,421 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:11,421 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:11,421 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:11,422 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:11,422 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:11,422 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:11,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:11,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:11,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:11,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:11,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:11,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:11,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:11,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:11,423 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:11,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:11,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:11,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:11,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,425 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:11,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:11,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,434 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:11,435 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:11,435 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:11,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:11,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:11,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:11,435 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:11,435 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:11,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:11,435 DEBUG [Thread-874] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:11,435 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:11,435 DEBUG [Thread-874] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:11,435 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,435 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:11,435 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:11,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:11,435 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:11,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:11,517 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=125, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/A/8ffad9d8b64f467ba18dd030b31ba473 2023-04-20 12:54:11,535 DEBUG [Thread-874] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:11,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:11,537 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=125 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/B/102c6975cb804f4793118ac9d21d74f8 2023-04-20 12:54:11,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:11,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:11,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:11,540 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:11,540 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:11,540 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:11,540 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:11,540 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:11,540 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:11,542 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:11,542 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:11,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:11,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:11,543 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,543 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:11,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:11,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:11,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:11,543 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:11,544 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:11,544 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:11,544 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:11,544 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:11,544 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:11,554 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:11,554 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:11,554 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:11,554 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. started... 2023-04-20 12:54:11,554 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. as already flushing 2023-04-20 12:54:11,554 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:11,554 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:11,554 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:11,554 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,554 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:11,554 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:11,563 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=125 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/C/08bde1af4d224a2ab8896a97fbc960be 2023-04-20 12:54:11,565 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,565 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:11,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:11,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:11,566 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:11,566 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:11,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:11,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:11,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:11,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:11,568 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/A/8ffad9d8b64f467ba18dd030b31ba473 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/8ffad9d8b64f467ba18dd030b31ba473 2023-04-20 12:54:11,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:11,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1367ab9a[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:11,569 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:11,569 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:11,572 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/8ffad9d8b64f467ba18dd030b31ba473, entries=200, sequenceid=125, filesize=31.9 K 2023-04-20 12:54:11,573 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/B/102c6975cb804f4793118ac9d21d74f8 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/102c6975cb804f4793118ac9d21d74f8 2023-04-20 12:54:11,573 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:11,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:11,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:11,574 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:11,574 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,574 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:11,574 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:11,574 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:11,575 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:11,575 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,575 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:11,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:11,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:11,575 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:11,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:11,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:11,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:11,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:11,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:11,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:11,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:11,578 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:11,578 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:11,578 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:11,578 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:11,578 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:11,579 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:11,579 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:11,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:11,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:11,579 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:11,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:11,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:11,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:11,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:11,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:11,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:11,580 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/102c6975cb804f4793118ac9d21d74f8, entries=150, sequenceid=125, filesize=11.6 K 2023-04-20 12:54:11,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:11,580 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:11,581 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/C/08bde1af4d224a2ab8896a97fbc960be as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/08bde1af4d224a2ab8896a97fbc960be 2023-04-20 12:54:11,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:11,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:11,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,582 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:11,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:11,590 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/08bde1af4d224a2ab8896a97fbc960be, entries=150, sequenceid=125, filesize=11.6 K 2023-04-20 12:54:11,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,591 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 4782eca030b67d2c82cb20827b0bb953 in 892ms, sequenceid=125, compaction requested=true 2023-04-20 12:54:11,591 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4782eca030b67d2c82cb20827b0bb953: 2023-04-20 12:54:11,591 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:11,591 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:54:11,592 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:54:11,591 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:54:11,592 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:54:11,592 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:11,592 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,592 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:11,592 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:11,592 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:11,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:11,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:11,592 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:11,592 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,592 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:11,592 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:11,592 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:11,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:11,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:11,592 DEBUG [Thread-874] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:11,593 DEBUG [Thread-874] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:11,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:11,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:11,593 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35796 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:54:11,593 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): 4782eca030b67d2c82cb20827b0bb953/B is initiating minor compaction (all files) 2023-04-20 12:54:11,593 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4782eca030b67d2c82cb20827b0bb953/B in TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:11,593 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/7c9780a9f8f842c09b2e5b444f6f33d6, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/1368f72210ee45e5bdbace57728fc31d, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/102c6975cb804f4793118ac9d21d74f8] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp, totalSize=35.0 K 2023-04-20 12:54:11,593 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84316 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:54:11,594 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): 4782eca030b67d2c82cb20827b0bb953/A is initiating minor compaction (all files) 2023-04-20 12:54:11,594 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4782eca030b67d2c82cb20827b0bb953/A in TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:11,594 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/ee0511af6fa54322a01dd6f4429cf8c3, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/80a7470001324a8d9fb0a7d41e86a431, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/8ffad9d8b64f467ba18dd030b31ba473] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp, totalSize=82.3 K 2023-04-20 12:54:11,594 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting ee0511af6fa54322a01dd6f4429cf8c3, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=74, earliestPutTs=1681995248404 2023-04-20 12:54:11,594 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 7c9780a9f8f842c09b2e5b444f6f33d6, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=74, earliestPutTs=1681995248404 2023-04-20 12:54:11,595 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 80a7470001324a8d9fb0a7d41e86a431, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1681995248475 2023-04-20 12:54:11,595 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 1368f72210ee45e5bdbace57728fc31d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1681995248475 2023-04-20 12:54:11,595 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 8ffad9d8b64f467ba18dd030b31ba473, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1681995249568 2023-04-20 12:54:11,595 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 102c6975cb804f4793118ac9d21d74f8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1681995249574 2023-04-20 12:54:11,603 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): 4782eca030b67d2c82cb20827b0bb953#B#compaction#128 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:11,616 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4782eca030b67d2c82cb20827b0bb953#A#compaction#127 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:11,635 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/B/e5f1f0e6fc7f488292bd33daeaa9baea as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/e5f1f0e6fc7f488292bd33daeaa9baea 2023-04-20 12:54:11,640 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4782eca030b67d2c82cb20827b0bb953/B of 4782eca030b67d2c82cb20827b0bb953 into e5f1f0e6fc7f488292bd33daeaa9baea(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:54:11,640 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4782eca030b67d2c82cb20827b0bb953: 2023-04-20 12:54:11,640 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953., storeName=4782eca030b67d2c82cb20827b0bb953/B, priority=13, startTime=1681995251591; duration=0sec 2023-04-20 12:54:11,640 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:54:11,640 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:54:11,641 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35796 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:54:11,641 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): 4782eca030b67d2c82cb20827b0bb953/C is initiating minor compaction (all files) 2023-04-20 12:54:11,641 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4782eca030b67d2c82cb20827b0bb953/C in TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:11,641 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/4c5eb47fd0fc45b592c4095368ef7946, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/bb39d0be8dfe41ae9cebe9cdd7286a09, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/08bde1af4d224a2ab8896a97fbc960be] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp, totalSize=35.0 K 2023-04-20 12:54:11,642 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 4c5eb47fd0fc45b592c4095368ef7946, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=74, earliestPutTs=1681995248404 2023-04-20 12:54:11,642 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting bb39d0be8dfe41ae9cebe9cdd7286a09, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1681995248475 2023-04-20 12:54:11,642 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 08bde1af4d224a2ab8896a97fbc960be, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1681995249574 2023-04-20 12:54:11,648 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): 4782eca030b67d2c82cb20827b0bb953#C#compaction#129 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:11,693 DEBUG [Thread-874] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:11,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:11,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:11,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:11,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:11,696 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:11,696 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:11,696 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:11,696 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:11,697 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:11,697 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:11,697 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,697 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:11,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:11,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:11,698 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,698 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:11,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:11,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:11,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:11,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:11,698 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:11,698 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:11,698 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:11,698 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:11,699 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:11,699 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:11,699 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:11,699 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:11,699 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:11,699 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. started... 2023-04-20 12:54:11,699 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4782eca030b67d2c82cb20827b0bb953 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-20 12:54:11,699 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=A 2023-04-20 12:54:11,699 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:11,699 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=B 2023-04-20 12:54:11,699 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:11,699 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=C 2023-04-20 12:54:11,699 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:11,713 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304209ec1662b1036427c8f912a8edaf1d08f to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304209ec1662b1036427c8f912a8edaf1d08f 2023-04-20 12:54:11,719 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=146, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/A/3e83fc685ac543ba8f603337e9d1814f 2023-04-20 12:54:11,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 4782eca030b67d2c82cb20827b0bb953 2023-04-20 12:54:11,850 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. as already flushing 2023-04-20 12:54:11,866 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:11,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34280 deadline: 1681995311864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,867 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:11,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34248 deadline: 1681995311866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,969 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:11,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34280 deadline: 1681995311967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:11,969 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:11,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34248 deadline: 1681995311968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,056 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/C/9576fe8fe1544d8cb52b845b37be9110 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/9576fe8fe1544d8cb52b845b37be9110 2023-04-20 12:54:12,061 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4782eca030b67d2c82cb20827b0bb953/C of 4782eca030b67d2c82cb20827b0bb953 into 9576fe8fe1544d8cb52b845b37be9110(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:54:12,061 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4782eca030b67d2c82cb20827b0bb953: 2023-04-20 12:54:12,061 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953., storeName=4782eca030b67d2c82cb20827b0bb953/C, priority=13, startTime=1681995251592; duration=0sec 2023-04-20 12:54:12,061 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:12,131 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/B/afa5b957f54340c49c5c58ea68ccd62a 2023-04-20 12:54:12,171 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:12,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34248 deadline: 1681995312170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,172 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:12,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34280 deadline: 1681995312171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,450 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/A/953933cdec0649a6b2136ae341bfeb8c as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/953933cdec0649a6b2136ae341bfeb8c 2023-04-20 12:54:12,457 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4782eca030b67d2c82cb20827b0bb953/A of 4782eca030b67d2c82cb20827b0bb953 into 953933cdec0649a6b2136ae341bfeb8c(size=25.4 K), total size for store is 25.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:54:12,457 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4782eca030b67d2c82cb20827b0bb953: 2023-04-20 12:54:12,457 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953., storeName=4782eca030b67d2c82cb20827b0bb953/A, priority=13, startTime=1681995251591; duration=0sec 2023-04-20 12:54:12,457 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:12,474 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:12,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34248 deadline: 1681995312472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,475 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:12,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34280 deadline: 1681995312474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,548 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/C/ed52df96adb34631a4e6b445e7d2d79b 2023-04-20 12:54:12,552 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/A/3e83fc685ac543ba8f603337e9d1814f as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/3e83fc685ac543ba8f603337e9d1814f 2023-04-20 12:54:12,556 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/3e83fc685ac543ba8f603337e9d1814f, entries=150, sequenceid=146, filesize=25.3 K 2023-04-20 12:54:12,557 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/B/afa5b957f54340c49c5c58ea68ccd62a as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/afa5b957f54340c49c5c58ea68ccd62a 2023-04-20 12:54:12,560 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/afa5b957f54340c49c5c58ea68ccd62a, entries=150, sequenceid=146, filesize=11.8 K 2023-04-20 12:54:12,561 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/C/ed52df96adb34631a4e6b445e7d2d79b as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/ed52df96adb34631a4e6b445e7d2d79b 2023-04-20 12:54:12,565 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/ed52df96adb34631a4e6b445e7d2d79b, entries=150, sequenceid=146, filesize=11.8 K 2023-04-20 12:54:12,565 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 4782eca030b67d2c82cb20827b0bb953 in 866ms, sequenceid=146, compaction requested=false 2023-04-20 12:54:12,566 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4782eca030b67d2c82cb20827b0bb953: 2023-04-20 12:54:12,566 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:12,566 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:12,566 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:12,566 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,566 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:12,566 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:12,567 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,567 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:12,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:12,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:12,567 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:12,567 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:12,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:12,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:12,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:12,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:12,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:12,568 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:12,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3da48878[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:12,568 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:12,569 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:12,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:12,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:12,569 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:12,569 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:12,569 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:12,569 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,569 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:12,570 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,570 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:12,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:12,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:12,570 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:12,570 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:12,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:12,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:12,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:12,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:12,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:12,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:12,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:12,573 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:12,573 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:12,573 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:12,573 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:12,573 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:12,574 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:12,574 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:12,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:12,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:12,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:12,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:12,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:12,575 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:12,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:12,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:12,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:12,575 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:12,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:12,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:12,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:12,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,576 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:12,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:12,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,578 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,578 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:12,578 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:12,578 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:12,578 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:12,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:12,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:12,578 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:12,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:12,578 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:12,578 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:12,578 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:12,578 DEBUG [Thread-874] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:12,578 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:12,578 DEBUG [Thread-874] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:12,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:12,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:12,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 4782eca030b67d2c82cb20827b0bb953 2023-04-20 12:54:12,610 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4782eca030b67d2c82cb20827b0bb953 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-20 12:54:12,610 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=A 2023-04-20 12:54:12,610 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:12,610 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=B 2023-04-20 12:54:12,610 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:12,610 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=C 2023-04-20 12:54:12,610 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:12,622 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304201496a3190554484299449d691e085eb7 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304201496a3190554484299449d691e085eb7 2023-04-20 12:54:12,627 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=161, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/A/9e4f8a0f5aad43af8c7a91b36d23c318 2023-04-20 12:54:12,637 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:12,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34222 deadline: 1681995312634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,637 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:12,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34232 deadline: 1681995312635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,678 DEBUG [Thread-874] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:12,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:12,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:12,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:12,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:12,681 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:12,681 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:12,681 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:12,681 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:12,682 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:12,682 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:12,683 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:12,683 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:12,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:12,683 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,683 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:12,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:12,683 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:12,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:12,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:12,683 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:12,684 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:12,684 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:12,684 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:12,684 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:12,684 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:12,684 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:12,684 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:12,684 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:12,684 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. started... 2023-04-20 12:54:12,684 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. as already flushing 2023-04-20 12:54:12,684 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:12,684 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:12,684 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:12,684 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,684 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:12,684 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:12,685 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:12,685 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:12,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:12,685 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:12,685 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:12,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:12,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:12,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:12,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:12,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:12,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@785c0004[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:12,686 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:12,686 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:12,687 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:12,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:12,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:12,687 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:12,687 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:12,687 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:12,687 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,687 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:12,688 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:12,688 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:12,688 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,688 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:12,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:12,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:12,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:12,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:12,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:12,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:12,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:12,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:12,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:12,690 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:12,690 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:12,690 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:12,690 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:12,690 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:12,691 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:12,691 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:12,695 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:12,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:12,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:12,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:12,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:12,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:12,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:12,695 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:12,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:12,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:12,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:12,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:12,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:12,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,696 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:12,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:12,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,698 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:12,698 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:12,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:12,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:12,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:12,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:12,699 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,699 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:12,699 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:12,699 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:12,699 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,699 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:12,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:12,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:12,699 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:12,699 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:12,700 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:12,700 DEBUG [Thread-874] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:12,700 DEBUG [Thread-874] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:12,740 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:12,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34222 deadline: 1681995312738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,741 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:12,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34232 deadline: 1681995312738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,800 DEBUG [Thread-874] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:12,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:12,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:12,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:12,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:12,803 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:12,803 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:12,803 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:12,803 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:12,803 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:12,803 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:12,805 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:12,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:12,805 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:12,805 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,805 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:12,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:12,805 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:12,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:12,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:12,806 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:12,806 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:12,806 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:12,806 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:12,806 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:12,806 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:12,806 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:12,806 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:12,806 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:12,806 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. started... 2023-04-20 12:54:12,806 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. as already flushing 2023-04-20 12:54:12,806 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:12,806 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:12,806 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:12,806 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,807 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:12,807 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:12,808 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,808 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:12,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:12,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:12,808 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:12,808 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:12,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:12,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:12,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:12,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:12,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:12,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a9d6c65[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:12,817 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:12,817 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:12,818 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:12,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:12,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:12,818 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:12,818 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:12,818 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,818 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:12,818 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:12,819 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:12,819 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,819 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:12,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:12,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:12,819 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:12,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:12,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:12,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:12,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:12,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:12,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:12,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:12,821 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:12,821 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:12,821 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:12,821 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:12,821 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:12,822 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:12,822 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:12,822 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:12,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:12,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:12,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:12,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:12,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:12,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:12,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:12,822 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:12,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:12,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:12,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:12,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:12,823 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:12,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:12,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,825 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:12,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:12,825 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:12,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:12,825 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:12,825 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:12,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:12,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:12,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:12,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:12,825 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:12,825 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:12,825 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,825 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:12,825 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:12,825 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:12,825 DEBUG [Thread-874] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:12,825 DEBUG [Thread-874] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:12,925 DEBUG [Thread-874] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:12,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:12,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:12,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:12,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:12,928 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:12,928 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:12,928 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:12,928 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:12,929 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:12,929 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:12,929 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:12,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:12,929 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:12,930 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,930 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:12,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:12,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:12,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:12,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:12,930 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:12,931 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:12,931 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:12,931 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:12,931 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:12,931 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:12,931 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:12,931 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:12,931 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. started... 2023-04-20 12:54:12,931 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. as already flushing 2023-04-20 12:54:12,931 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:12,931 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:12,931 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:12,931 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,931 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:12,931 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:12,932 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:12,932 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:12,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:12,932 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:12,932 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:12,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:12,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:12,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:12,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:12,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:12,933 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:12,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@640825d7[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:12,933 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:12,934 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:12,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:12,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:12,934 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:12,934 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:12,934 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:12,934 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,934 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:12,935 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:12,935 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,935 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:12,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:12,935 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:12,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:12,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:12,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:12,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:12,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:12,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:12,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:12,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:12,936 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:12,937 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:12,937 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:12,937 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:12,937 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:12,937 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:12,937 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:12,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:12,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:12,937 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:12,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:12,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:12,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:12,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:12,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:12,938 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:12,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:12,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:12,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:12,939 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:12,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:12,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:12,940 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,940 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:12,940 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:12,940 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:12,940 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:12,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:12,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:12,940 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:12,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:12,940 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:12,940 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:12,940 DEBUG [Thread-874] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:12,940 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:12,940 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:12,940 DEBUG [Thread-874] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:12,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:12,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:12,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:12,946 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:12,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34222 deadline: 1681995312942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,946 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:12,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34232 deadline: 1681995312942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,976 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:12,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34280 deadline: 1681995312976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:12,977 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:12,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34248 deadline: 1681995312976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,038 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=161 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/B/282c1242f05d477e8dbc838c113c3c15 2023-04-20 12:54:13,041 DEBUG [Thread-874] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:13,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:13,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:13,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:13,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:13,045 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:13,045 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:13,045 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:13,045 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:13,045 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:13,045 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:13,046 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:13,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:13,046 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:13,046 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,046 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:13,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:13,046 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:13,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:13,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:13,047 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:13,047 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:13,047 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:13,047 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:13,047 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:13,047 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:13,048 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:13,048 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:13,048 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:13,048 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. started... 2023-04-20 12:54:13,048 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. as already flushing 2023-04-20 12:54:13,048 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:13,048 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:13,048 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:13,048 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,048 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:13,048 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:13,049 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,049 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:13,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:13,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:13,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:13,049 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:13,049 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:13,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:13,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:13,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:13,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:13,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7df69666[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:13,050 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:13,051 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:13,051 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:13,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:13,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:13,051 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:13,051 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:13,051 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:13,051 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,051 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:13,052 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,052 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:13,052 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:13,052 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:13,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:13,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:13,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:13,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:13,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:13,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:13,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:13,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:13,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:13,054 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:13,054 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:13,054 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:13,054 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:13,054 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:13,055 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:13,055 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:13,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:13,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:13,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:13,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:13,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:13,055 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:13,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:13,055 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:13,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:13,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:13,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:13,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:13,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:13,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,056 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:13,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:13,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,058 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,058 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:13,058 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:13,058 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,058 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:13,058 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:13,058 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:13,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:13,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:13,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:13,058 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:13,058 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:13,058 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:13,058 DEBUG [Thread-874] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:13,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:13,058 DEBUG [Thread-874] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:13,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:13,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:13,159 DEBUG [Thread-874] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:13,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:13,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:13,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:13,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:13,161 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:13,161 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:13,161 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:13,162 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:13,162 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:13,162 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:13,162 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:13,162 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:13,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:13,163 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,163 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:13,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:13,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:13,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:13,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:13,163 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:13,164 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:13,164 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:13,164 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:13,164 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:13,164 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:13,164 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:13,164 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:13,164 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:13,164 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. started... 2023-04-20 12:54:13,164 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. as already flushing 2023-04-20 12:54:13,164 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:13,164 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:13,164 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:13,164 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,164 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:13,164 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:13,166 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,166 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:13,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:13,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:13,166 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:13,166 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:13,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:13,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:13,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:13,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:13,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:13,167 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:13,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6532f5fc[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:13,167 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:13,167 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:13,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:13,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:13,167 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:13,168 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:13,168 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:13,168 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,168 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:13,168 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:13,168 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,168 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:13,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:13,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:13,168 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:13,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:13,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:13,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:13,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:13,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:13,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:13,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:13,170 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:13,170 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:13,170 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:13,170 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:13,170 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:13,171 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:13,171 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:13,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:13,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:13,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:13,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:13,171 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:13,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:13,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:13,172 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:13,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:13,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:13,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:13,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:13,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:13,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,173 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:13,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:13,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,174 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,174 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:13,174 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:13,174 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,174 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:13,174 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:13,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:13,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:13,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:13,174 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:13,174 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:13,175 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:13,175 DEBUG [Thread-874] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:13,175 DEBUG [Thread-874] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:13,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:13,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:13,175 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:13,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:13,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:13,249 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:13,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34232 deadline: 1681995313247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,249 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:13,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34222 deadline: 1681995313249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,275 DEBUG [Thread-874] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:13,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:13,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:13,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:13,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:13,278 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:13,278 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:13,278 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:13,278 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:13,278 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:13,278 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:13,279 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:13,279 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:13,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:13,279 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,279 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:13,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:13,279 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:13,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:13,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:13,280 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:13,280 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:13,280 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:13,280 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:13,280 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:13,280 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:13,280 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:13,280 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:13,280 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:13,280 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. started... 2023-04-20 12:54:13,281 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. as already flushing 2023-04-20 12:54:13,281 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:13,281 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:13,281 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:13,281 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,281 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:13,281 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:13,281 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:13,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:13,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:13,282 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:13,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:13,282 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:13,282 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:13,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:13,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:13,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:13,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@446d437b[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:13,283 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:13,283 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:13,283 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:13,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:13,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:13,284 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:13,284 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:13,284 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:13,284 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,284 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:13,284 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,284 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:13,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:13,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:13,284 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:13,284 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:13,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:13,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:13,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:13,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:13,286 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:13,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:13,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34254 deadline: 1681995313284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,287 DEBUG [Thread-866] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8156 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953., hostname=jenkins-hbase20.apache.org,36277,1681995213947, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-20 12:54:13,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:13,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:13,287 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:13,287 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:13,288 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:13,288 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:13,288 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:13,288 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:13,288 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:13,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:13,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:13,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:13,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:13,288 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:13,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:13,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:13,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:13,289 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:13,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:13,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:13,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:13,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:13,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,290 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:13,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:13,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,291 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:13,291 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:13,291 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:13,291 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:13,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:13,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:13,292 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:13,291 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:13,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:13,292 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,292 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:13,292 DEBUG [Thread-874] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:13,292 DEBUG [Thread-874] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:13,292 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:13,292 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:13,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:13,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:13,392 DEBUG [Thread-874] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:13,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:13,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:13,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:13,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:13,395 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:13,395 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:13,395 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:13,395 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:13,395 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:13,395 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:13,396 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,396 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:13,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:13,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:13,396 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,396 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:13,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:13,397 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:13,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:13,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:13,397 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:13,397 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:13,397 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:13,397 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:13,397 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:13,397 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:13,398 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:13,398 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:13,398 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:13,398 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. started... 2023-04-20 12:54:13,398 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. as already flushing 2023-04-20 12:54:13,398 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:13,398 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:13,398 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:13,398 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,398 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:13,398 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:13,399 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,399 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:13,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:13,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:13,399 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:13,399 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:13,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:13,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:13,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:13,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:13,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:13,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61eee0bc[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:13,400 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:13,400 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:13,401 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:13,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:13,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:13,401 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:13,401 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:13,401 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:13,401 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,401 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:13,402 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,402 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:13,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:13,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:13,402 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:13,402 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:13,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:13,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:13,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:13,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:13,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:13,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:13,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:13,404 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:13,404 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:13,404 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:13,404 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:13,404 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:13,404 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:13,405 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:13,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:13,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:13,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:13,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:13,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:13,405 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:13,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:13,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:13,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:13,405 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:13,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:13,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:13,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:13,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,406 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:13,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:13,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,408 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:13,408 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:13,408 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:13,408 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:13,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:13,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:13,409 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:13,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:13,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:13,409 DEBUG [Thread-874] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:13,409 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:13,409 DEBUG [Thread-874] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:13,409 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,409 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:13,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:13,409 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:13,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:13,409 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:13,449 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=161 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/C/e2e71db2727745a2abf4a9d38bfa9ef3 2023-04-20 12:54:13,454 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/A/9e4f8a0f5aad43af8c7a91b36d23c318 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/9e4f8a0f5aad43af8c7a91b36d23c318 2023-04-20 12:54:13,457 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/9e4f8a0f5aad43af8c7a91b36d23c318, entries=200, sequenceid=161, filesize=32.1 K 2023-04-20 12:54:13,459 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/B/282c1242f05d477e8dbc838c113c3c15 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/282c1242f05d477e8dbc838c113c3c15 2023-04-20 12:54:13,464 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/282c1242f05d477e8dbc838c113c3c15, entries=150, sequenceid=161, filesize=11.8 K 2023-04-20 12:54:13,465 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/C/e2e71db2727745a2abf4a9d38bfa9ef3 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/e2e71db2727745a2abf4a9d38bfa9ef3 2023-04-20 12:54:13,469 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/e2e71db2727745a2abf4a9d38bfa9ef3, entries=150, sequenceid=161, filesize=11.8 K 2023-04-20 12:54:13,470 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 4782eca030b67d2c82cb20827b0bb953 in 861ms, sequenceid=161, compaction requested=true 2023-04-20 12:54:13,470 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4782eca030b67d2c82cb20827b0bb953: 2023-04-20 12:54:13,470 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:13,470 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:54:13,471 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:54:13,470 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:54:13,471 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:54:13,483 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84768 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:54:13,483 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): 4782eca030b67d2c82cb20827b0bb953/A is initiating minor compaction (all files) 2023-04-20 12:54:13,483 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36198 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:54:13,483 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4782eca030b67d2c82cb20827b0bb953/A in TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:13,483 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): 4782eca030b67d2c82cb20827b0bb953/B is initiating minor compaction (all files) 2023-04-20 12:54:13,483 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4782eca030b67d2c82cb20827b0bb953/B in TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:13,483 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/953933cdec0649a6b2136ae341bfeb8c, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/3e83fc685ac543ba8f603337e9d1814f, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/9e4f8a0f5aad43af8c7a91b36d23c318] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp, totalSize=82.8 K 2023-04-20 12:54:13,483 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/e5f1f0e6fc7f488292bd33daeaa9baea, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/afa5b957f54340c49c5c58ea68ccd62a, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/282c1242f05d477e8dbc838c113c3c15] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp, totalSize=35.3 K 2023-04-20 12:54:13,484 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting e5f1f0e6fc7f488292bd33daeaa9baea, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1681995249574 2023-04-20 12:54:13,484 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 953933cdec0649a6b2136ae341bfeb8c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1681995249574 2023-04-20 12:54:13,484 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting afa5b957f54340c49c5c58ea68ccd62a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1681995250731 2023-04-20 12:54:13,484 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 3e83fc685ac543ba8f603337e9d1814f, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1681995250731 2023-04-20 12:54:13,484 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 282c1242f05d477e8dbc838c113c3c15, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1681995251857 2023-04-20 12:54:13,485 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 9e4f8a0f5aad43af8c7a91b36d23c318, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1681995251857 2023-04-20 12:54:13,495 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): 4782eca030b67d2c82cb20827b0bb953#B#compaction#136 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:13,507 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4782eca030b67d2c82cb20827b0bb953#A#compaction#137 average throughput is 1.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:13,509 DEBUG [Thread-874] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:13,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:13,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:13,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:13,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:13,513 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:13,513 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:13,513 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:13,513 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:13,513 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:13,513 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:13,514 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:13,514 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:13,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:13,514 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,514 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:13,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:13,515 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:13,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:13,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:13,515 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:13,515 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:13,515 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:13,515 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:13,515 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:13,515 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:13,515 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:13,515 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:13,515 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:13,515 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. started... 2023-04-20 12:54:13,516 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4782eca030b67d2c82cb20827b0bb953 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-20 12:54:13,516 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=A 2023-04-20 12:54:13,516 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:13,516 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=B 2023-04-20 12:54:13,516 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:13,516 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=C 2023-04-20 12:54:13,516 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:13,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 4782eca030b67d2c82cb20827b0bb953 2023-04-20 12:54:13,754 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. as already flushing 2023-04-20 12:54:13,793 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:13,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34222 deadline: 1681995313792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,793 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:13,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34232 deadline: 1681995313793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,894 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:13,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34222 deadline: 1681995313894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,894 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:13,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34232 deadline: 1681995313894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,909 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/B/013fa588f6f845858f4cd78002c7fe2c as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/013fa588f6f845858f4cd78002c7fe2c 2023-04-20 12:54:13,913 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4782eca030b67d2c82cb20827b0bb953/B of 4782eca030b67d2c82cb20827b0bb953 into 013fa588f6f845858f4cd78002c7fe2c(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:54:13,913 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4782eca030b67d2c82cb20827b0bb953: 2023-04-20 12:54:13,913 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953., storeName=4782eca030b67d2c82cb20827b0bb953/B, priority=13, startTime=1681995253470; duration=0sec 2023-04-20 12:54:13,913 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:54:13,913 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:54:13,917 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36198 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:54:13,917 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): 4782eca030b67d2c82cb20827b0bb953/C is initiating minor compaction (all files) 2023-04-20 12:54:13,917 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4782eca030b67d2c82cb20827b0bb953/C in TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:13,917 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/9576fe8fe1544d8cb52b845b37be9110, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/ed52df96adb34631a4e6b445e7d2d79b, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/e2e71db2727745a2abf4a9d38bfa9ef3] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp, totalSize=35.3 K 2023-04-20 12:54:13,919 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 9576fe8fe1544d8cb52b845b37be9110, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1681995249574 2023-04-20 12:54:13,919 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting ed52df96adb34631a4e6b445e7d2d79b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1681995250731 2023-04-20 12:54:13,920 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting e2e71db2727745a2abf4a9d38bfa9ef3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1681995251857 2023-04-20 12:54:13,929 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): 4782eca030b67d2c82cb20827b0bb953#C#compaction#139 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:13,931 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420511563ada8054c5b9c7cf23c56f3923d to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420511563ada8054c5b9c7cf23c56f3923d 2023-04-20 12:54:13,975 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/A/6aa1e2a7360f490ea4cba1362dfaedd7 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/6aa1e2a7360f490ea4cba1362dfaedd7 2023-04-20 12:54:13,979 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4782eca030b67d2c82cb20827b0bb953/A of 4782eca030b67d2c82cb20827b0bb953 into 6aa1e2a7360f490ea4cba1362dfaedd7(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:54:13,979 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4782eca030b67d2c82cb20827b0bb953: 2023-04-20 12:54:13,979 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953., storeName=4782eca030b67d2c82cb20827b0bb953/A, priority=13, startTime=1681995253470; duration=0sec 2023-04-20 12:54:13,979 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:13,980 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:13,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34248 deadline: 1681995313979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:13,982 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:13,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34280 deadline: 1681995313982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:14,097 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:14,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34222 deadline: 1681995314095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:14,097 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:14,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34232 deadline: 1681995314095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:14,373 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=180, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/A/e11e38b0331d4c2c89f977039fa63c60 2023-04-20 12:54:14,385 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/C/f227800671794e118b2b0c2a48bf91c2 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/f227800671794e118b2b0c2a48bf91c2 2023-04-20 12:54:14,390 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 4782eca030b67d2c82cb20827b0bb953/C of 4782eca030b67d2c82cb20827b0bb953 into f227800671794e118b2b0c2a48bf91c2(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:54:14,390 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4782eca030b67d2c82cb20827b0bb953: 2023-04-20 12:54:14,390 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953., storeName=4782eca030b67d2c82cb20827b0bb953/C, priority=13, startTime=1681995253471; duration=0sec 2023-04-20 12:54:14,390 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:14,399 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:14,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34232 deadline: 1681995314398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:14,400 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:14,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34222 deadline: 1681995314400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:14,811 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=180 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/B/745e830ea18042d1b3243041d1d34066 2023-04-20 12:54:14,902 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:14,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34222 deadline: 1681995314901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:14,903 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:14,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34232 deadline: 1681995314901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:15,221 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=180 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/C/19208133e2bb420eaff8f660750f199e 2023-04-20 12:54:15,225 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/A/e11e38b0331d4c2c89f977039fa63c60 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/e11e38b0331d4c2c89f977039fa63c60 2023-04-20 12:54:15,229 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/e11e38b0331d4c2c89f977039fa63c60, entries=150, sequenceid=180, filesize=25.3 K 2023-04-20 12:54:15,230 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/B/745e830ea18042d1b3243041d1d34066 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/745e830ea18042d1b3243041d1d34066 2023-04-20 12:54:15,238 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/745e830ea18042d1b3243041d1d34066, entries=150, sequenceid=180, filesize=11.8 K 2023-04-20 12:54:15,239 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/C/19208133e2bb420eaff8f660750f199e as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/19208133e2bb420eaff8f660750f199e 2023-04-20 12:54:15,244 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/19208133e2bb420eaff8f660750f199e, entries=150, sequenceid=180, filesize=11.8 K 2023-04-20 12:54:15,245 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 4782eca030b67d2c82cb20827b0bb953 in 1729ms, sequenceid=180, compaction requested=false 2023-04-20 12:54:15,245 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4782eca030b67d2c82cb20827b0bb953: 2023-04-20 12:54:15,245 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:15,245 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:15,245 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:15,245 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:15,245 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:15,245 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:15,246 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:15,246 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:15,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:15,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:15,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:15,247 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:15,247 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:15,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:15,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:15,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:15,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:15,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:15,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:15,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28e20f41[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:15,248 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:15,248 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:15,248 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:15,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:15,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:15,249 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:15,249 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:15,249 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:15,249 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:15,249 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:15,250 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:15,250 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:15,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:15,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:15,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:15,250 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:15,251 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:15,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:15,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:15,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:15,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:15,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:15,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:15,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:15,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:15,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:15,252 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:15,252 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:15,252 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:15,252 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:15,252 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:15,253 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:15,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:15,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:15,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:15,253 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:15,254 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:15,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:15,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:15,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:15,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:15,254 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:15,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:15,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:15,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:15,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:15,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:15,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:15,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:15,255 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:15,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:15,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:15,257 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:15,257 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:15,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:15,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:15,257 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:15,257 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:15,257 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:15,257 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:15,257 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:15,257 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:15,257 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:15,257 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:15,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:15,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:15,258 DEBUG [Thread-874] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:15,258 DEBUG [Thread-874] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:15,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:15,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:15,358 DEBUG [Thread-874] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:15,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:15,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:15,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:15,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:15,361 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:15,361 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:15,361 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:15,361 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:15,361 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:15,361 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:15,362 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:15,362 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:15,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:15,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:15,362 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:15,362 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:15,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:15,362 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:15,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:15,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:15,362 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:15,363 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:15,363 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:15,363 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:15,363 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:15,363 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:15,363 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:15,363 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:15,363 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:15,363 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. started... 2023-04-20 12:54:15,363 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4782eca030b67d2c82cb20827b0bb953 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-20 12:54:15,364 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=A 2023-04-20 12:54:15,364 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:15,364 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=B 2023-04-20 12:54:15,364 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:15,364 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=C 2023-04-20 12:54:15,364 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:15,377 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304209de7a642dda54b7bacc027da8336d1f4 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304209de7a642dda54b7bacc027da8336d1f4 2023-04-20 12:54:15,782 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=196, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/A/2ab79fbd86814dd286824b3780c7ec66 2023-04-20 12:54:15,812 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=196 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/B/158ae6e4135c404f97249c408b53540c 2023-04-20 12:54:15,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 4782eca030b67d2c82cb20827b0bb953 2023-04-20 12:54:15,909 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. as already flushing 2023-04-20 12:54:15,933 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:15,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34222 deadline: 1681995315931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:15,934 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:15,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34232 deadline: 1681995315931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:15,987 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:15,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34248 deadline: 1681995315987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:15,989 DEBUG [Thread-868] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4122 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953., hostname=jenkins-hbase20.apache.org,36277,1681995213947, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-20 12:54:15,993 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:15,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34280 deadline: 1681995315991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:15,995 DEBUG [Thread-870] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4130 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953., hostname=jenkins-hbase20.apache.org,36277,1681995213947, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-20 12:54:16,036 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:16,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34222 deadline: 1681995316034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:16,036 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:16,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34232 deadline: 1681995316035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:16,226 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=196 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/C/a86610b1e68849dc8a72f18d440c7eb0 2023-04-20 12:54:16,230 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/A/2ab79fbd86814dd286824b3780c7ec66 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/2ab79fbd86814dd286824b3780c7ec66 2023-04-20 12:54:16,234 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/2ab79fbd86814dd286824b3780c7ec66, entries=150, sequenceid=196, filesize=25.3 K 2023-04-20 12:54:16,235 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/B/158ae6e4135c404f97249c408b53540c as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/158ae6e4135c404f97249c408b53540c 2023-04-20 12:54:16,239 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/158ae6e4135c404f97249c408b53540c, entries=150, sequenceid=196, filesize=11.8 K 2023-04-20 12:54:16,239 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:16,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34222 deadline: 1681995316238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:16,239 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:16,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34232 deadline: 1681995316238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:16,240 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/C/a86610b1e68849dc8a72f18d440c7eb0 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/a86610b1e68849dc8a72f18d440c7eb0 2023-04-20 12:54:16,244 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/a86610b1e68849dc8a72f18d440c7eb0, entries=150, sequenceid=196, filesize=11.8 K 2023-04-20 12:54:16,244 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 4782eca030b67d2c82cb20827b0bb953 in 881ms, sequenceid=196, compaction requested=true 2023-04-20 12:54:16,244 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4782eca030b67d2c82cb20827b0bb953: 2023-04-20 12:54:16,244 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:16,244 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:16,244 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:16,245 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:16,245 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:16,245 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:16,246 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:16,246 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:16,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:16,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:16,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:16,246 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:16,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:16,246 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:16,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:16,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:16,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:16,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:16,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:16,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a8d82b4[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:16,247 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:16,247 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:16,248 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:16,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:16,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:16,248 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:16,248 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:16,248 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:16,248 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:16,248 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:16,249 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:16,249 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:16,249 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:16,249 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:16,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:16,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:16,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:16,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:16,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:16,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:16,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:16,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:16,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:16,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:16,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:16,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:16,251 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:16,251 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:16,251 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:16,251 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:16,251 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:16,252 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:16,252 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:16,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:16,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:16,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:16,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:16,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:16,252 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:16,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:16,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:16,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:16,253 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:16,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:16,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:16,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:16,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:16,254 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:16,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:16,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:16,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:16,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:16,255 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:16,255 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:16,255 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:16,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:16,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:16,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:16,255 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:16,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:16,255 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:16,255 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:16,255 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:16,255 DEBUG [Thread-874] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:16,255 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:16,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:16,256 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:16,256 DEBUG [Thread-874] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:16,256 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:16,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:16,356 DEBUG [Thread-874] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:16,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:16,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:16,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:16,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:16,358 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:16,358 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:16,358 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:16,358 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:16,359 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:16,359 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:16,359 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:16,359 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:16,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:16,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:16,360 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:16,360 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:16,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:16,360 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:16,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:16,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:16,360 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:16,360 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:16,360 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:16,360 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:16,361 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:16,361 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:16,361 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:16,361 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:16,361 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:16,361 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. started... 2023-04-20 12:54:16,361 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4782eca030b67d2c82cb20827b0bb953 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-20 12:54:16,361 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=A 2023-04-20 12:54:16,361 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:16,362 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=B 2023-04-20 12:54:16,362 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:16,362 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=C 2023-04-20 12:54:16,362 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:16,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 4782eca030b67d2c82cb20827b0bb953 2023-04-20 12:54:16,543 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. as already flushing 2023-04-20 12:54:16,555 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:16,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34222 deadline: 1681995316553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:16,558 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:16,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34232 deadline: 1681995316555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:16,657 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:16,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34222 deadline: 1681995316656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:16,660 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:16,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34232 deadline: 1681995316659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:16,779 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304204a82725e5e834e2fba9536d1b37bfc9b to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304204a82725e5e834e2fba9536d1b37bfc9b 2023-04-20 12:54:16,785 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=215, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/A/a2016c140eaf4520946713c14959ca41 2023-04-20 12:54:16,799 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=215 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/B/f433965d9d4e457e89750ce4e6e6094c 2023-04-20 12:54:16,860 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:16,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34222 deadline: 1681995316858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:16,862 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:16,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34232 deadline: 1681995316862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:17,163 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:17,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34222 deadline: 1681995317162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:17,164 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:17,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34232 deadline: 1681995317163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:17,210 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=215 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/C/1efa56be7ea146a1a80af03065044ec1 2023-04-20 12:54:17,214 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/A/a2016c140eaf4520946713c14959ca41 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/a2016c140eaf4520946713c14959ca41 2023-04-20 12:54:17,218 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/a2016c140eaf4520946713c14959ca41, entries=150, sequenceid=215, filesize=25.3 K 2023-04-20 12:54:17,219 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/B/f433965d9d4e457e89750ce4e6e6094c as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/f433965d9d4e457e89750ce4e6e6094c 2023-04-20 12:54:17,223 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/f433965d9d4e457e89750ce4e6e6094c, entries=150, sequenceid=215, filesize=11.8 K 2023-04-20 12:54:17,223 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/C/1efa56be7ea146a1a80af03065044ec1 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/1efa56be7ea146a1a80af03065044ec1 2023-04-20 12:54:17,227 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/1efa56be7ea146a1a80af03065044ec1, entries=150, sequenceid=215, filesize=11.8 K 2023-04-20 12:54:17,227 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 4782eca030b67d2c82cb20827b0bb953 in 866ms, sequenceid=215, compaction requested=true 2023-04-20 12:54:17,227 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4782eca030b67d2c82cb20827b0bb953: 2023-04-20 12:54:17,227 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:17,227 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:17,228 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:17,228 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:17,228 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:17,228 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:17,228 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:17,228 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:17,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:17,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:17,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:17,229 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:17,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:17,229 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:17,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:17,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:17,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:17,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:17,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:17,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74557405[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:17,230 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:17,230 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:17,231 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:17,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:17,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:17,231 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:17,231 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:17,231 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:17,231 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:17,231 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:17,232 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:17,232 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:17,232 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:17,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:17,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:17,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:17,232 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:17,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:17,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:17,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:17,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:17,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:17,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:17,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:17,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:17,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:17,233 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:17,233 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:17,233 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:17,233 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:17,233 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:17,235 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:17,236 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:17,237 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:17,237 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:17,237 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:17,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:17,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:17,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:17,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:17,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:17,237 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:17,238 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:17,238 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:17,237 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:17,238 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:17,238 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:17,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:17,238 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:17,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:17,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:17,238 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:17,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:17,238 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:17,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:17,238 DEBUG [Thread-874] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:17,238 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:17,238 DEBUG [Thread-874] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:17,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:17,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:17,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:17,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:17,239 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:17,338 DEBUG [Thread-874] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:17,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:17,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:17,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:17,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:17,341 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:17,341 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:17,341 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:17,341 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:17,341 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:17,341 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:17,342 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:17,342 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:17,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:17,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:17,342 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:17,342 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:17,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:17,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:17,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:17,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:17,343 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:17,343 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:17,343 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:17,343 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:17,343 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:17,343 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:17,343 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:17,343 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:17,343 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:17,343 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. started... 2023-04-20 12:54:17,344 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4782eca030b67d2c82cb20827b0bb953 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-20 12:54:17,344 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=A 2023-04-20 12:54:17,344 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:17,344 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=B 2023-04-20 12:54:17,344 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:17,344 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=C 2023-04-20 12:54:17,344 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:17,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 4782eca030b67d2c82cb20827b0bb953 2023-04-20 12:54:17,670 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. as already flushing 2023-04-20 12:54:17,699 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:17,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34232 deadline: 1681995317697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:17,700 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:17,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34222 deadline: 1681995317697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:17,760 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420dc67d52fb94341c9a047218455278eec to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420dc67d52fb94341c9a047218455278eec 2023-04-20 12:54:17,802 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:17,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34232 deadline: 1681995317800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:17,803 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:17,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34222 deadline: 1681995317801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:18,006 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:18,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34222 deadline: 1681995318003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:18,007 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:18,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34232 deadline: 1681995318004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:18,164 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=228, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/A/779ebbb96df74457ad62c5f0f409db36 2023-04-20 12:54:18,177 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=228 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/B/f354b6cfaae84ace8ad9250dc12e0040 2023-04-20 12:54:18,310 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:18,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34222 deadline: 1681995318309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:18,310 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:18,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34232 deadline: 1681995318309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:18,594 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=228 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/C/40c1ba9d1f484f9693bcf44bf1aafb67 2023-04-20 12:54:18,599 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/A/779ebbb96df74457ad62c5f0f409db36 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/779ebbb96df74457ad62c5f0f409db36 2023-04-20 12:54:18,602 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/779ebbb96df74457ad62c5f0f409db36, entries=150, sequenceid=228, filesize=25.3 K 2023-04-20 12:54:18,603 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/B/f354b6cfaae84ace8ad9250dc12e0040 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/f354b6cfaae84ace8ad9250dc12e0040 2023-04-20 12:54:18,607 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/f354b6cfaae84ace8ad9250dc12e0040, entries=150, sequenceid=228, filesize=11.8 K 2023-04-20 12:54:18,608 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/C/40c1ba9d1f484f9693bcf44bf1aafb67 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/40c1ba9d1f484f9693bcf44bf1aafb67 2023-04-20 12:54:18,612 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/40c1ba9d1f484f9693bcf44bf1aafb67, entries=150, sequenceid=228, filesize=11.8 K 2023-04-20 12:54:18,612 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 4782eca030b67d2c82cb20827b0bb953 in 1268ms, sequenceid=228, compaction requested=true 2023-04-20 12:54:18,612 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4782eca030b67d2c82cb20827b0bb953: 2023-04-20 12:54:18,612 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:18,612 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:18,612 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:18,612 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:18,612 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:18,613 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:18,613 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:18,613 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:18,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:18,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:18,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:18,614 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:18,614 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:18,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:18,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:18,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:18,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:18,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:18,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:18,615 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:18,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62a89133[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:18,615 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:18,615 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:18,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:18,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:18,616 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:18,616 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:18,616 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:18,616 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:18,616 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:18,616 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:18,616 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:18,616 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:18,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:18,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:18,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:18,616 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:18,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:18,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:18,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:18,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:18,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:18,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:18,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:18,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:18,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:18,618 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:18,618 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:18,618 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:18,618 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:18,618 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:18,620 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:18,620 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:18,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:18,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:18,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:18,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:18,620 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:18,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:18,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:18,621 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:18,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:18,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:18,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:18,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:18,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:18,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:18,621 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:18,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:18,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:18,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:18,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:18,623 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:18,623 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:18,623 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:18,623 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:18,623 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:18,623 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:18,623 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:18,623 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:18,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:18,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:18,623 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:18,623 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:18,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:18,623 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:18,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:18,623 DEBUG [Thread-874] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:18,623 DEBUG [Thread-874] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:18,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:18,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:18,723 DEBUG [Thread-874] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:18,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:18,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:18,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:18,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:18,726 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:18,727 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:18,727 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:18,727 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:18,727 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:18,727 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:18,727 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:18,727 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:18,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:18,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:18,728 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:18,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:18,728 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:18,728 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:18,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:18,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:18,728 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:18,728 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:18,728 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:18,729 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:18,729 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:18,729 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:18,729 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:18,729 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:18,729 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:18,729 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. started... 2023-04-20 12:54:18,729 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4782eca030b67d2c82cb20827b0bb953 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-20 12:54:18,729 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=A 2023-04-20 12:54:18,729 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:18,729 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=B 2023-04-20 12:54:18,729 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:18,729 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=C 2023-04-20 12:54:18,729 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:18,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 4782eca030b67d2c82cb20827b0bb953 2023-04-20 12:54:18,813 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. as already flushing 2023-04-20 12:54:18,826 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:18,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34232 deadline: 1681995318823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:18,826 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:18,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34222 deadline: 1681995318825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:18,928 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:18,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34232 deadline: 1681995318927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:18,928 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:18,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34222 deadline: 1681995318927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:19,131 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:19,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34232 deadline: 1681995319129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:19,131 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:19,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34222 deadline: 1681995319129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:19,143 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420a3f445728b8f406d8323fba90e08bb14 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420a3f445728b8f406d8323fba90e08bb14 2023-04-20 12:54:19,434 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:19,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34232 deadline: 1681995319432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:19,435 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:19,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34222 deadline: 1681995319434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:19,548 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=247, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/A/d593e7b0d6ca435da28e77a19d1793f8 2023-04-20 12:54:19,558 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=247 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/B/a15b85e13a164b698b1dd9bd256aee66 2023-04-20 12:54:19,936 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:19,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34232 deadline: 1681995319936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:19,942 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:19,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34222 deadline: 1681995319939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:19,971 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=247 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/C/f0e701d1368147e694a623ec38f3bee7 2023-04-20 12:54:19,975 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/A/d593e7b0d6ca435da28e77a19d1793f8 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/d593e7b0d6ca435da28e77a19d1793f8 2023-04-20 12:54:19,979 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/d593e7b0d6ca435da28e77a19d1793f8, entries=150, sequenceid=247, filesize=25.3 K 2023-04-20 12:54:19,980 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/B/a15b85e13a164b698b1dd9bd256aee66 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/a15b85e13a164b698b1dd9bd256aee66 2023-04-20 12:54:19,985 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/a15b85e13a164b698b1dd9bd256aee66, entries=150, sequenceid=247, filesize=11.8 K 2023-04-20 12:54:19,994 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/C/f0e701d1368147e694a623ec38f3bee7 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/f0e701d1368147e694a623ec38f3bee7 2023-04-20 12:54:19,998 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/f0e701d1368147e694a623ec38f3bee7, entries=150, sequenceid=247, filesize=11.8 K 2023-04-20 12:54:19,999 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 4782eca030b67d2c82cb20827b0bb953 in 1269ms, sequenceid=247, compaction requested=true 2023-04-20 12:54:19,999 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4782eca030b67d2c82cb20827b0bb953: 2023-04-20 12:54:19,999 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:19,999 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:19,999 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:19,999 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:19,999 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:19,999 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:20,000 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,000 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:20,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:20,000 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,000 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:20,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:20,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:20,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:20,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:20,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:20,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a080220[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:20,001 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:20,001 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,002 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,002 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:20,002 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:20,002 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:20,002 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,002 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:20,003 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:20,003 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,003 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:20,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:20,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:20,003 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:20,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:20,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:20,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:20,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:20,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:20,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:20,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:20,004 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:20,005 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:20,005 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:20,005 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:20,005 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:20,005 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,005 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:20,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:20,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,005 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:20,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:20,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:20,006 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:20,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:20,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:20,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:20,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,007 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:20,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:20,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 4782eca030b67d2c82cb20827b0bb953 2023-04-20 12:54:20,012 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4782eca030b67d2c82cb20827b0bb953 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-20 12:54:20,012 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=A 2023-04-20 12:54:20,012 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:20,012 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=B 2023-04-20 12:54:20,012 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:20,012 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=C 2023-04-20 12:54:20,012 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:20,014 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,014 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:20,014 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:20,014 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:20,014 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:20,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:20,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:20,015 DEBUG [Thread-874] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:20,014 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,014 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:20,015 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,015 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,015 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,015 DEBUG [Thread-874] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:20,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:20,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:20,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:20,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:20,042 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:20,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34248 deadline: 1681995320036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,042 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:20,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34280 deadline: 1681995320036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,115 DEBUG [Thread-874] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:20,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:20,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:20,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:20,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:20,118 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:20,118 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:20,118 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:20,118 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:20,118 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,118 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:20,119 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:20,119 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:20,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:20,119 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,119 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:20,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:20,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:20,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:20,120 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:20,120 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:20,120 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:20,120 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:20,120 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:20,120 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:20,120 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:20,120 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:20,120 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:20,120 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. started... 2023-04-20 12:54:20,121 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. as already flushing 2023-04-20 12:54:20,121 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:20,121 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:20,121 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:20,121 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,121 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:20,121 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:20,121 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,121 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:20,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:20,122 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,122 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:20,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:20,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:20,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:20,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:20,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:20,123 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:20,123 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c26b3f7[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:20,123 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,123 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:20,123 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:20,123 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:20,124 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,124 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:20,124 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:20,124 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,124 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:20,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:20,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:20,124 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:20,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:20,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:20,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:20,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:20,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:20,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:20,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:20,126 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:20,126 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:20,126 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:20,126 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:20,126 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:20,127 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,127 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:20,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:20,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,127 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:20,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:20,127 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:20,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:20,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:20,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:20,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:20,128 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:20,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:20,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,129 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,129 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:20,129 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:20,129 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:20,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:20,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:20,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:20,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:20,129 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:20,129 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:20,130 DEBUG [Thread-874] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:20,130 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,130 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,130 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:20,130 DEBUG [Thread-874] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:20,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:20,130 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:20,146 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:20,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34248 deadline: 1681995320143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,146 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:20,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34280 deadline: 1681995320143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,230 DEBUG [Thread-874] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:20,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:20,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:20,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:20,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:20,233 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:20,233 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:20,233 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:20,233 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:20,233 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,233 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:20,234 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:20,234 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:20,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:20,234 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,234 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:20,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:20,235 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:20,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:20,235 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:20,235 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:20,235 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:20,235 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:20,235 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:20,235 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:20,235 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:20,236 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:20,236 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:20,236 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. started... 2023-04-20 12:54:20,236 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. as already flushing 2023-04-20 12:54:20,236 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:20,236 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:20,236 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:20,236 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,236 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:20,236 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:20,237 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,237 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:20,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:20,237 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,237 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:20,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:20,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:20,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:20,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:20,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:20,238 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:20,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@330c072a[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:20,238 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,239 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,239 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:20,239 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:20,239 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,239 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:20,239 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:20,239 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,240 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:20,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:20,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:20,240 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:20,240 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:20,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:20,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:20,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:20,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:20,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:20,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:20,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:20,244 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:20,244 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:20,244 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:20,244 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:20,244 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:20,248 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,248 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:20,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:20,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,248 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:20,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:20,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:20,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:20,249 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:20,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:20,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:20,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,250 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:20,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:20,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,252 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,253 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:20,253 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:20,253 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:20,253 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:20,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:20,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:20,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:20,253 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:20,253 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:20,253 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,253 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,253 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:20,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:20,253 DEBUG [Thread-874] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:20,254 DEBUG [Thread-874] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:20,350 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:20,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34248 deadline: 1681995320348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,350 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:20,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34280 deadline: 1681995320348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,354 DEBUG [Thread-874] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:20,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:20,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:20,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:20,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:20,356 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:20,357 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:20,357 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:20,357 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:20,357 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,357 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:20,358 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,358 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:20,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:20,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:20,358 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,358 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:20,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:20,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:20,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:20,359 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:20,359 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:20,359 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:20,359 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:20,359 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:20,359 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:20,359 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:20,359 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:20,359 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:20,359 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. started... 2023-04-20 12:54:20,359 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. as already flushing 2023-04-20 12:54:20,359 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:20,359 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:20,359 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:20,359 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,359 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:20,360 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:20,360 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,360 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:20,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:20,361 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,361 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:20,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:20,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:20,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:20,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:20,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:20,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@150b2971[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:20,362 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:20,362 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,362 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,362 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:20,362 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:20,362 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,363 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:20,363 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:20,363 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:20,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:20,363 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:20,363 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:20,363 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:20,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:20,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:20,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:20,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:20,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:20,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:20,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:20,365 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:20,365 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:20,365 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:20,365 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:20,365 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:20,366 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,366 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:20,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,366 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:20,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:20,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:20,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:20,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:20,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,367 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:20,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:20,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:20,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,368 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:20,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:20,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,370 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,370 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:20,370 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:20,370 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:20,370 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:20,370 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:20,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:20,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:20,370 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:20,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:20,370 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,371 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,371 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,371 DEBUG [Thread-874] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:20,371 DEBUG [Thread-874] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:20,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:20,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:20,425 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042050a71cfcfc3a4cec9c98add7c31db99e to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042050a71cfcfc3a4cec9c98add7c31db99e 2023-04-20 12:54:20,431 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=262, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/A/da93fe7491904832840b44ed958d5c90 2023-04-20 12:54:20,471 DEBUG [Thread-874] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:20,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:20,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:20,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:20,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:20,474 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:20,474 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:20,474 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:20,474 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:20,475 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,475 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:20,475 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:20,475 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:20,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:20,476 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,476 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:20,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:20,476 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:20,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:20,476 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:20,476 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:20,477 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:20,477 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:20,477 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:20,477 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:20,477 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:20,477 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:20,477 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:20,477 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. started... 2023-04-20 12:54:20,477 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. as already flushing 2023-04-20 12:54:20,477 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:20,477 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:20,477 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:20,477 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,477 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:20,477 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:20,478 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,478 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:20,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:20,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:20,478 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,478 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:20,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:20,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:20,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:20,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:20,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f0830f2[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:20,479 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:20,479 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,480 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,480 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,480 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:20,480 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:20,480 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:20,480 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:20,481 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,481 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:20,481 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:20,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:20,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:20,481 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:20,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:20,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:20,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:20,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:20,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:20,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:20,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:20,482 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:20,482 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:20,483 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:20,483 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:20,483 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:20,483 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,483 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:20,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:20,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,483 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:20,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:20,484 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:20,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:20,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:20,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:20,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:20,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:20,486 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:20,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,487 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,487 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:20,487 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:20,487 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:20,487 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,487 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:20,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:20,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:20,487 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:20,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:20,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:20,487 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,488 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:20,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:20,488 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,488 DEBUG [Thread-874] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:20,488 DEBUG [Thread-874] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:20,588 DEBUG [Thread-874] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:20,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:20,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:20,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:20,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:20,591 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:20,591 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:20,591 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:20,591 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:20,591 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,591 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:20,592 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:20,592 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:20,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:20,592 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,592 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:20,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:20,593 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:20,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:20,593 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:20,593 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:20,593 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:20,593 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:20,593 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:20,593 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:20,594 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:20,594 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:20,594 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:20,594 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. started... 2023-04-20 12:54:20,594 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. as already flushing 2023-04-20 12:54:20,594 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:20,594 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:20,594 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:20,594 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,594 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:20,594 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:20,595 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,595 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:20,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:20,595 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,595 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:20,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:20,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:20,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:20,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:20,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:20,597 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:20,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@566883ef[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:20,597 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,597 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,597 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:20,597 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:20,598 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:20,598 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,598 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:20,602 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,602 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:20,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:20,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:20,602 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:20,602 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:20,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:20,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:20,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:20,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:20,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:20,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:20,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:20,605 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:20,605 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:20,605 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:20,605 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:20,605 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:20,606 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,606 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:20,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:20,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,606 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:20,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:20,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:20,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:20,607 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:20,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:20,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:20,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,608 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:20,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:20,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,610 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,610 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:20,610 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:20,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:20,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:20,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:20,610 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:20,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:20,611 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:20,610 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:20,611 DEBUG [Thread-874] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:20,610 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:20,611 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,611 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,611 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,611 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,611 DEBUG [Thread-874] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:20,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:20,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:20,653 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:20,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34280 deadline: 1681995320652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,656 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:20,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34248 deadline: 1681995320653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,711 DEBUG [Thread-874] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:20,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:20,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:20,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:20,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:20,718 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:20,718 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:20,718 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:20,718 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:20,719 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,719 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:20,720 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,720 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:20,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:20,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:20,721 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,721 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:20,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:20,721 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:20,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:20,722 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:20,722 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:20,722 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:20,722 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:20,722 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:20,722 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:20,722 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:20,722 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:20,723 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:20,723 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. started... 2023-04-20 12:54:20,726 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. as already flushing 2023-04-20 12:54:20,726 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:20,726 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:20,726 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:20,726 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,726 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:20,726 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:20,739 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,740 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:20,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:20,740 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,740 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:20,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:20,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:20,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:20,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:20,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:20,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50efb5f[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:20,742 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:20,742 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,749 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,749 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:20,749 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:20,749 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,749 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:20,749 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:20,756 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,756 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:20,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:20,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:20,756 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:20,756 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:20,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:20,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:20,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:20,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:20,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:20,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:20,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:20,759 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:20,759 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:20,759 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:20,759 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:20,759 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:20,765 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,765 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,766 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:20,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:20,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:20,766 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:20,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:20,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:20,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:20,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:20,767 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:20,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:20,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:20,776 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,776 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:20,776 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:20,776 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,776 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,776 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,776 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:20,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:20,776 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:20,777 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:20,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:20,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:20,776 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:20,777 DEBUG [Thread-874] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:20,777 DEBUG [Thread-874] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:20,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:20,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:20,777 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:20,847 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=262 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/B/e5bd8e8ef651491f80d07684c5206967 2023-04-20 12:54:20,877 DEBUG [Thread-874] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:20,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:20,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:20,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:20,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:20,880 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:20,881 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:20,881 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:20,881 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:20,881 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=262 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/C/9d91e996788048a4a58153e762996a3f 2023-04-20 12:54:20,881 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,881 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:20,882 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:20,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:20,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:20,882 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,882 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,882 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:20,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:20,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:20,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:20,883 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:20,887 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/A/da93fe7491904832840b44ed958d5c90 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/da93fe7491904832840b44ed958d5c90 2023-04-20 12:54:20,891 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:20,891 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:20,891 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:20,891 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:20,891 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:20,893 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/da93fe7491904832840b44ed958d5c90, entries=250, sequenceid=262, filesize=39.1 K 2023-04-20 12:54:20,894 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/B/e5bd8e8ef651491f80d07684c5206967 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/e5bd8e8ef651491f80d07684c5206967 2023-04-20 12:54:20,899 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/e5bd8e8ef651491f80d07684c5206967, entries=150, sequenceid=262, filesize=11.9 K 2023-04-20 12:54:20,900 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/C/9d91e996788048a4a58153e762996a3f as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/9d91e996788048a4a58153e762996a3f 2023-04-20 12:54:20,902 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:20,902 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:20,902 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:20,902 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. started... 2023-04-20 12:54:20,902 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. as already flushing 2023-04-20 12:54:20,902 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:20,902 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:20,902 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:20,902 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,906 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:20,906 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:20,907 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,907 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:20,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:20,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:20,908 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,908 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:20,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:20,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:20,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:20,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:20,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49020a2e[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:20,909 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:20,909 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,910 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,910 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:20,910 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:20,910 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:20,911 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:20,911 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:20,911 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:20,911 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,911 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:20,911 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:20,911 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:20,911 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:20,911 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:20,911 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:20,911 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:20,913 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,913 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:20,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:20,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,913 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:20,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:20,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:20,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:20,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:20,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,913 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:20,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:20,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:20,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:20,916 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,917 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,917 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:20,917 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:20,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:20,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:20,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:20,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:20,917 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:20,917 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:20,917 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:20,918 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,918 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,918 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:20,918 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:20,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:20,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:20,918 DEBUG [Thread-874] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:20,918 DEBUG [Thread-874] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:20,928 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/9d91e996788048a4a58153e762996a3f, entries=150, sequenceid=262, filesize=11.9 K 2023-04-20 12:54:20,929 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 4782eca030b67d2c82cb20827b0bb953 in 918ms, sequenceid=262, compaction requested=true 2023-04-20 12:54:20,929 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4782eca030b67d2c82cb20827b0bb953: 2023-04-20 12:54:20,930 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:20,930 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-20 12:54:20,930 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-20 12:54:20,930 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:20,931 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:54:20,940 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 84744 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-20 12:54:20,940 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): 4782eca030b67d2c82cb20827b0bb953/B is initiating minor compaction (all files) 2023-04-20 12:54:20,940 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4782eca030b67d2c82cb20827b0bb953/B in TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:20,940 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/013fa588f6f845858f4cd78002c7fe2c, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/745e830ea18042d1b3243041d1d34066, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/158ae6e4135c404f97249c408b53540c, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/f433965d9d4e457e89750ce4e6e6094c, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/f354b6cfaae84ace8ad9250dc12e0040, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/a15b85e13a164b698b1dd9bd256aee66, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/e5bd8e8ef651491f80d07684c5206967] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp, totalSize=82.8 K 2023-04-20 12:54:20,941 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 013fa588f6f845858f4cd78002c7fe2c, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1681995251857 2023-04-20 12:54:20,943 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 745e830ea18042d1b3243041d1d34066, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1681995252626 2023-04-20 12:54:20,944 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 158ae6e4135c404f97249c408b53540c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1681995253786 2023-04-20 12:54:20,945 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting f433965d9d4e457e89750ce4e6e6094c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1681995255927 2023-04-20 12:54:20,945 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting f354b6cfaae84ace8ad9250dc12e0040, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1681995256547 2023-04-20 12:54:20,945 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting a15b85e13a164b698b1dd9bd256aee66, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1681995257690 2023-04-20 12:54:20,946 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting e5bd8e8ef651491f80d07684c5206967, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1681995258822 2023-04-20 12:54:20,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 4782eca030b67d2c82cb20827b0bb953 2023-04-20 12:54:20,950 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4782eca030b67d2c82cb20827b0bb953 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-20 12:54:20,950 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=A 2023-04-20 12:54:20,950 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:20,950 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=B 2023-04-20 12:54:20,951 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:20,951 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=C 2023-04-20 12:54:20,951 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:20,954 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 195798 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-20 12:54:20,954 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): 4782eca030b67d2c82cb20827b0bb953/A is initiating minor compaction (all files) 2023-04-20 12:54:20,954 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4782eca030b67d2c82cb20827b0bb953/A in TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:20,954 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/6aa1e2a7360f490ea4cba1362dfaedd7, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/e11e38b0331d4c2c89f977039fa63c60, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/2ab79fbd86814dd286824b3780c7ec66, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/a2016c140eaf4520946713c14959ca41, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/779ebbb96df74457ad62c5f0f409db36, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/d593e7b0d6ca435da28e77a19d1793f8, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/da93fe7491904832840b44ed958d5c90] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp, totalSize=191.2 K 2023-04-20 12:54:20,955 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 6aa1e2a7360f490ea4cba1362dfaedd7, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1681995251857 2023-04-20 12:54:20,957 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting e11e38b0331d4c2c89f977039fa63c60, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1681995252626 2023-04-20 12:54:20,958 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 2ab79fbd86814dd286824b3780c7ec66, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1681995253786 2023-04-20 12:54:20,958 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting a2016c140eaf4520946713c14959ca41, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1681995255927 2023-04-20 12:54:20,959 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 779ebbb96df74457ad62c5f0f409db36, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1681995256547 2023-04-20 12:54:20,960 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting d593e7b0d6ca435da28e77a19d1793f8, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1681995257690 2023-04-20 12:54:20,961 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting da93fe7491904832840b44ed958d5c90, keycount=250, bloomtype=ROW, size=39.1 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1681995258818 2023-04-20 12:54:20,968 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): 4782eca030b67d2c82cb20827b0bb953#B#compaction#157 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:20,992 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:20,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34222 deadline: 1681995320984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:20,998 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:20,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34232 deadline: 1681995320992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,010 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420d39b9704507e4e15ae585c62fc57b041 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420d39b9704507e4e15ae585c62fc57b041 2023-04-20 12:54:21,018 DEBUG [Thread-874] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:21,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:21,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:21,020 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4782eca030b67d2c82cb20827b0bb953#A#compaction#159 average throughput is 0.90 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:21,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:21,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:21,046 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:21,046 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:21,046 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:21,046 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:21,047 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:21,047 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:21,048 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:21,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:21,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:21,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:21,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:21,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:21,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:21,049 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:21,050 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:21,050 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:21,050 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:21,050 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:21,050 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:21,051 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,051 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,052 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:21,054 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=280, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/A/3945b1c82cd042cfbc5275a33bd6495c 2023-04-20 12:54:21,062 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:21,062 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:21,066 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:21,066 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. started... 2023-04-20 12:54:21,066 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. as already flushing 2023-04-20 12:54:21,066 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:21,066 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:21,066 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:21,066 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,068 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:21,068 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:21,069 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,069 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:21,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:21,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:21,070 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:21,070 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:21,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:21,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:21,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:21,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:21,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:21,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c33c5c[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:21,071 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:21,071 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:21,072 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:21,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:21,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:21,073 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:21,073 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:21,073 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:21,073 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,073 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:21,073 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:21,073 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:21,074 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,074 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:21,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:21,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:21,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:21,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:21,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:21,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:21,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:21,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:21,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:21,075 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:21,075 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:21,075 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:21,075 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:21,075 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:21,076 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:21,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:21,076 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:21,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:21,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:21,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:21,076 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:21,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:21,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:21,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:21,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:21,077 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:21,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:21,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:21,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:21,078 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:21,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:21,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,079 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:21,079 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:21,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:21,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:21,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:21,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:21,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:21,080 DEBUG [Thread-874] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:21,080 DEBUG [Thread-874] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:21,079 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,080 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:21,080 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:21,080 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,080 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:21,080 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:21,080 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:21,079 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:21,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:21,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:21,085 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=280 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/B/9b0e00cb5aed4423969ebf15ea82d6f7 2023-04-20 12:54:21,101 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:21,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34222 deadline: 1681995321093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,104 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:21,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34232 deadline: 1681995321100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,162 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:21,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34280 deadline: 1681995321157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,162 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:21,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34248 deadline: 1681995321161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,180 DEBUG [Thread-874] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:21,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:21,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:21,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:21,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:21,182 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:21,182 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:21,182 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:21,182 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:21,183 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:21,183 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:21,183 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:21,183 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:21,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:21,184 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,184 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:21,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:21,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:21,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:21,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:21,184 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:21,184 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:21,184 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:21,184 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:21,184 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:21,184 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:21,185 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:21,185 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:21,185 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:21,185 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. started... 2023-04-20 12:54:21,185 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. as already flushing 2023-04-20 12:54:21,185 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:21,185 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:21,185 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:21,185 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,185 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:21,185 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:21,187 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,187 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:21,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:21,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:21,187 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:21,187 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:21,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:21,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:21,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:21,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:21,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:21,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1526a0b0[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:21,188 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:21,189 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:21,190 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:21,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:21,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:21,190 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:21,190 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:21,190 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:21,190 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,190 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:21,192 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:21,192 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:21,192 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:21,193 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:21,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:21,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:21,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:21,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:21,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:21,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:21,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:21,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:21,197 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:21,197 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:21,197 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:21,197 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:21,197 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:21,199 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:21,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:21,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:21,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:21,199 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:21,199 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:21,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:21,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:21,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:21,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:21,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:21,200 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:21,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:21,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:21,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:21,201 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:21,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:21,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,202 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:21,202 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,202 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:21,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:21,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:21,202 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:21,202 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:21,202 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:21,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:21,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:21,202 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:21,203 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,203 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:21,203 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:21,203 DEBUG [Thread-874] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:21,203 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:21,203 DEBUG [Thread-874] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:21,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:21,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:21,303 DEBUG [Thread-874] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:21,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:21,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:21,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:21,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:21,305 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:21,306 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:21,306 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:21,306 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:21,306 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:21,306 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:21,307 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,307 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:21,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:21,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:21,307 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,307 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:21,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:21,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:21,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:21,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:21,307 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:21,308 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:21,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34222 deadline: 1681995321302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,308 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:21,308 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:21,308 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:21,308 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:21,308 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:21,308 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:21,308 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:21,308 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:21,308 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. started... 2023-04-20 12:54:21,308 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. as already flushing 2023-04-20 12:54:21,308 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:21,308 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:21,308 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:21,308 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,308 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:21,308 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:21,309 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:21,309 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:21,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:21,309 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:21,309 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:21,309 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:21,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:21,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34232 deadline: 1681995321305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:21,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:21,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:21,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:21,311 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:21,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67007548[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:21,311 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:21,311 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:21,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:21,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:21,312 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:21,312 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:21,312 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:21,312 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,312 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:21,312 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:21,312 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,312 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:21,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:21,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:21,312 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:21,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:21,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:21,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:21,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:21,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:21,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:21,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:21,314 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:21,314 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:21,314 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:21,314 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:21,314 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:21,315 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:21,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:21,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:21,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:21,322 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:21,322 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:21,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:21,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:21,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:21,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:21,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:21,322 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:21,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:21,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:21,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:21,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,324 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:21,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:21,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,325 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,325 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:21,325 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:21,325 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:21,325 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:21,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:21,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:21,325 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:21,325 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:21,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:21,325 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:21,325 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:21,325 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:21,326 DEBUG [Thread-874] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:21,326 DEBUG [Thread-874] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:21,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:21,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:21,426 DEBUG [Thread-874] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:21,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:21,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:21,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:21,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:21,428 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:21,428 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:21,428 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:21,428 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:21,429 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:21,429 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:21,431 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:21,431 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:21,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:21,431 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,431 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:21,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:21,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:21,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:21,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:21,432 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:21,432 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:21,432 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:21,432 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:21,432 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:21,432 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:21,432 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:21,432 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:21,432 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:21,432 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. started... 2023-04-20 12:54:21,432 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. as already flushing 2023-04-20 12:54:21,432 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:21,432 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:21,432 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:21,432 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,433 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:21,433 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:21,433 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,433 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:21,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:21,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:21,434 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:21,434 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:21,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:21,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:21,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:21,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:21,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:21,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e8a40fc[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:21,435 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:21,435 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:21,436 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:21,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:21,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:21,436 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:21,436 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:21,436 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:21,436 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,436 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:21,437 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:21,437 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:21,437 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,437 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:21,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:21,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:21,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:21,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:21,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:21,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:21,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:21,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:21,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:21,439 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:21,439 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:21,439 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:21,439 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:21,439 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:21,440 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:21,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:21,440 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:21,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:21,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:21,440 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:21,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:21,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:21,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:21,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:21,441 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:21,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:21,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:21,442 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:21,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:21,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:21,443 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:21,443 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:21,443 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,443 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:21,444 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:21,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:21,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:21,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:21,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:21,444 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,444 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:21,444 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:21,444 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:21,444 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:21,444 DEBUG [Thread-874] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:21,444 DEBUG [Thread-874] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:21,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:21,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:21,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:21,452 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/B/47232aee7d4049408cf66cfd668cc3e0 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/47232aee7d4049408cf66cfd668cc3e0 2023-04-20 12:54:21,457 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 4782eca030b67d2c82cb20827b0bb953/B of 4782eca030b67d2c82cb20827b0bb953 into 47232aee7d4049408cf66cfd668cc3e0(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:54:21,457 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4782eca030b67d2c82cb20827b0bb953: 2023-04-20 12:54:21,457 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953., storeName=4782eca030b67d2c82cb20827b0bb953/B, priority=9, startTime=1681995260930; duration=0sec 2023-04-20 12:54:21,457 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:54:21,457 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-20 12:54:21,459 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 84744 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-20 12:54:21,459 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): 4782eca030b67d2c82cb20827b0bb953/C is initiating minor compaction (all files) 2023-04-20 12:54:21,459 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4782eca030b67d2c82cb20827b0bb953/C in TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:21,459 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/f227800671794e118b2b0c2a48bf91c2, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/19208133e2bb420eaff8f660750f199e, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/a86610b1e68849dc8a72f18d440c7eb0, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/1efa56be7ea146a1a80af03065044ec1, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/40c1ba9d1f484f9693bcf44bf1aafb67, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/f0e701d1368147e694a623ec38f3bee7, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/9d91e996788048a4a58153e762996a3f] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp, totalSize=82.8 K 2023-04-20 12:54:21,460 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting f227800671794e118b2b0c2a48bf91c2, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1681995251857 2023-04-20 12:54:21,460 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 19208133e2bb420eaff8f660750f199e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1681995252626 2023-04-20 12:54:21,460 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting a86610b1e68849dc8a72f18d440c7eb0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1681995253786 2023-04-20 12:54:21,461 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 1efa56be7ea146a1a80af03065044ec1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1681995255927 2023-04-20 12:54:21,461 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 40c1ba9d1f484f9693bcf44bf1aafb67, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1681995256547 2023-04-20 12:54:21,462 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting f0e701d1368147e694a623ec38f3bee7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1681995257690 2023-04-20 12:54:21,462 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 9d91e996788048a4a58153e762996a3f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1681995258822 2023-04-20 12:54:21,473 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): 4782eca030b67d2c82cb20827b0bb953#C#compaction#162 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:21,485 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/C/a4d32e8990e9476696bf214f29b269f8 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/a4d32e8990e9476696bf214f29b269f8 2023-04-20 12:54:21,500 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 4782eca030b67d2c82cb20827b0bb953/C of 4782eca030b67d2c82cb20827b0bb953 into a4d32e8990e9476696bf214f29b269f8(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:54:21,500 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4782eca030b67d2c82cb20827b0bb953: 2023-04-20 12:54:21,500 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=280 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/C/20495ac3ed144710adae3420a75e8911 2023-04-20 12:54:21,500 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953., storeName=4782eca030b67d2c82cb20827b0bb953/C, priority=9, startTime=1681995260931; duration=0sec 2023-04-20 12:54:21,500 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:21,504 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/A/3945b1c82cd042cfbc5275a33bd6495c as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/3945b1c82cd042cfbc5275a33bd6495c 2023-04-20 12:54:21,508 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/3945b1c82cd042cfbc5275a33bd6495c, entries=200, sequenceid=280, filesize=32.3 K 2023-04-20 12:54:21,510 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/B/9b0e00cb5aed4423969ebf15ea82d6f7 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/9b0e00cb5aed4423969ebf15ea82d6f7 2023-04-20 12:54:21,514 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/9b0e00cb5aed4423969ebf15ea82d6f7, entries=150, sequenceid=280, filesize=11.9 K 2023-04-20 12:54:21,515 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/C/20495ac3ed144710adae3420a75e8911 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/20495ac3ed144710adae3420a75e8911 2023-04-20 12:54:21,520 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/20495ac3ed144710adae3420a75e8911, entries=150, sequenceid=280, filesize=11.9 K 2023-04-20 12:54:21,521 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 4782eca030b67d2c82cb20827b0bb953 in 571ms, sequenceid=280, compaction requested=false 2023-04-20 12:54:21,521 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4782eca030b67d2c82cb20827b0bb953: 2023-04-20 12:54:21,544 DEBUG [Thread-874] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:21,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:21,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:21,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:21,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:21,547 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:21,547 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:21,547 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:21,547 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:21,547 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:21,548 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:21,548 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,548 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:21,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:21,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:21,548 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,548 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:21,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:21,549 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:21,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:21,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:21,549 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:21,549 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:21,549 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:21,549 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:21,549 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:21,549 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:21,549 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:21,549 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:21,549 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:21,549 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. started... 2023-04-20 12:54:21,550 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4782eca030b67d2c82cb20827b0bb953 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-20 12:54:21,550 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=A 2023-04-20 12:54:21,550 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:21,550 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=B 2023-04-20 12:54:21,550 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:21,550 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=C 2023-04-20 12:54:21,550 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:21,564 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304203ba05751ba7440678e88e88a7606ce83 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304203ba05751ba7440678e88e88a7606ce83 2023-04-20 12:54:21,585 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=296, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/A/9fe79b8a8c184656bee9fa36fc34a207 2023-04-20 12:54:21,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 4782eca030b67d2c82cb20827b0bb953 2023-04-20 12:54:21,613 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. as already flushing 2023-04-20 12:54:21,635 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:21,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34232 deadline: 1681995321631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,636 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:21,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34222 deadline: 1681995321631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,738 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:21,739 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:21,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34232 deadline: 1681995321736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34222 deadline: 1681995321737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,943 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:21,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34232 deadline: 1681995321941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,943 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:21,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34222 deadline: 1681995321941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:21,996 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=296 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/B/448b4afea6874b9fb777621d2356213e 2023-04-20 12:54:22,164 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:22,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34280 deadline: 1681995322163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:22,172 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:22,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34248 deadline: 1681995322169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:22,245 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:22,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34222 deadline: 1681995322244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:22,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:22,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34232 deadline: 1681995322245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:22,276 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/A/90131e57d1d34aaba02b201577e6b555 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/90131e57d1d34aaba02b201577e6b555 2023-04-20 12:54:22,280 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 4782eca030b67d2c82cb20827b0bb953/A of 4782eca030b67d2c82cb20827b0bb953 into 90131e57d1d34aaba02b201577e6b555(size=26.0 K), total size for store is 58.3 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-20 12:54:22,280 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4782eca030b67d2c82cb20827b0bb953: 2023-04-20 12:54:22,280 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953., storeName=4782eca030b67d2c82cb20827b0bb953/A, priority=9, startTime=1681995260930; duration=1sec 2023-04-20 12:54:22,281 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:22,405 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=296 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/C/b612fb0fc3fb4110af33cbd5d0385ea8 2023-04-20 12:54:22,410 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/A/9fe79b8a8c184656bee9fa36fc34a207 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/9fe79b8a8c184656bee9fa36fc34a207 2023-04-20 12:54:22,414 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/9fe79b8a8c184656bee9fa36fc34a207, entries=150, sequenceid=296, filesize=25.4 K 2023-04-20 12:54:22,415 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/B/448b4afea6874b9fb777621d2356213e as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/448b4afea6874b9fb777621d2356213e 2023-04-20 12:54:22,419 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/448b4afea6874b9fb777621d2356213e, entries=150, sequenceid=296, filesize=11.9 K 2023-04-20 12:54:22,420 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/C/b612fb0fc3fb4110af33cbd5d0385ea8 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/b612fb0fc3fb4110af33cbd5d0385ea8 2023-04-20 12:54:22,424 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/b612fb0fc3fb4110af33cbd5d0385ea8, entries=150, sequenceid=296, filesize=11.9 K 2023-04-20 12:54:22,425 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 4782eca030b67d2c82cb20827b0bb953 in 875ms, sequenceid=296, compaction requested=true 2023-04-20 12:54:22,425 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4782eca030b67d2c82cb20827b0bb953: 2023-04-20 12:54:22,425 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:22,425 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:22,426 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:22,426 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:22,426 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:22,426 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:22,427 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:22,427 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:22,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:22,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:22,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:22,427 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:22,427 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:22,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:22,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:22,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:22,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:22,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:22,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:22,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2035baad[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:22,428 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:22,428 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:22,430 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:22,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:22,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:22,430 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:22,430 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:22,430 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:22,431 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:22,431 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:22,431 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:22,431 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:22,431 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:22,431 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:22,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:22,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:22,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:22,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:22,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:22,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:22,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:22,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:22,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:22,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:22,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:22,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:22,433 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:22,433 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:22,433 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:22,433 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:22,434 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:22,434 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:22,434 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:22,434 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:22,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:22,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:22,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:22,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:22,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:22,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:22,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:22,435 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:22,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:22,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:22,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:22,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:22,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:22,436 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:22,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:22,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:22,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:22,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:22,438 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:22,438 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:22,438 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:22,438 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:22,438 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:22,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:22,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:22,438 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:22,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:22,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:22,438 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:22,438 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:22,438 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:22,438 DEBUG [Thread-874] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:22,438 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:22,439 DEBUG [Thread-874] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:22,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:22,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:22,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:22,539 DEBUG [Thread-874] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:22,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:22,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:22,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:22,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:22,541 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:22,541 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:22,541 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:22,542 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:22,542 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:22,542 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:22,542 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:22,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:22,543 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:22,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:22,543 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:22,543 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:22,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:22,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:22,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:22,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:22,543 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:22,544 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:22,544 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:22,544 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:22,544 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:22,544 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:22,544 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:22,544 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:22,544 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:22,545 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. started... 2023-04-20 12:54:22,545 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4782eca030b67d2c82cb20827b0bb953 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-20 12:54:22,545 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=A 2023-04-20 12:54:22,545 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:22,545 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=B 2023-04-20 12:54:22,545 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:22,545 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=C 2023-04-20 12:54:22,545 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:22,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 4782eca030b67d2c82cb20827b0bb953 2023-04-20 12:54:22,750 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. as already flushing 2023-04-20 12:54:22,761 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:22,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34232 deadline: 1681995322760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:22,763 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:22,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34222 deadline: 1681995322761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:22,863 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:22,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34232 deadline: 1681995322862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:22,865 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:22,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34222 deadline: 1681995322864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:22,960 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420c7c681349f614e18ac77c7270dc27182 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420c7c681349f614e18ac77c7270dc27182 2023-04-20 12:54:23,072 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:23,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34222 deadline: 1681995323069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:23,072 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:23,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34232 deadline: 1681995323069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:23,365 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=316, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/A/7ea719d1eff746b093b63365218c91c4 2023-04-20 12:54:23,366 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:23,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34254 deadline: 1681995323365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:23,368 DEBUG [Thread-866] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18237 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953., hostname=jenkins-hbase20.apache.org,36277,1681995213947, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-20 12:54:23,375 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:23,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34222 deadline: 1681995323374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:23,377 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:23,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34232 deadline: 1681995323376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:23,776 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=316 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/B/ae0b2e43456148cfa1c37d7921457f3a 2023-04-20 12:54:23,880 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:23,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34222 deadline: 1681995323877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:23,881 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:23,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34232 deadline: 1681995323879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:24,177 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:24,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34248 deadline: 1681995324175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:24,178 DEBUG [Thread-868] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4142 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953., hostname=jenkins-hbase20.apache.org,36277,1681995213947, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-20 12:54:24,182 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:24,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:34280 deadline: 1681995324178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:24,183 DEBUG [Thread-870] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4146 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953., hostname=jenkins-hbase20.apache.org,36277,1681995213947, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-20 12:54:24,185 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=316 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/C/2cd335cad36c44cabbd29aa1387990a2 2023-04-20 12:54:24,189 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/A/7ea719d1eff746b093b63365218c91c4 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/7ea719d1eff746b093b63365218c91c4 2023-04-20 12:54:24,194 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/7ea719d1eff746b093b63365218c91c4, entries=150, sequenceid=316, filesize=25.4 K 2023-04-20 12:54:24,194 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/B/ae0b2e43456148cfa1c37d7921457f3a as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/ae0b2e43456148cfa1c37d7921457f3a 2023-04-20 12:54:24,202 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/ae0b2e43456148cfa1c37d7921457f3a, entries=150, sequenceid=316, filesize=11.9 K 2023-04-20 12:54:24,203 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/C/2cd335cad36c44cabbd29aa1387990a2 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/2cd335cad36c44cabbd29aa1387990a2 2023-04-20 12:54:24,207 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/2cd335cad36c44cabbd29aa1387990a2, entries=150, sequenceid=316, filesize=11.9 K 2023-04-20 12:54:24,208 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 4782eca030b67d2c82cb20827b0bb953 in 1663ms, sequenceid=316, compaction requested=true 2023-04-20 12:54:24,208 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4782eca030b67d2c82cb20827b0bb953: 2023-04-20 12:54:24,208 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:24,208 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:24,208 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:24,208 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:24,208 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:24,208 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:24,209 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:24,209 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:24,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:24,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:24,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:24,210 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:24,210 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:24,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:24,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:24,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:24,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:24,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:24,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:24,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36ad3b[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:24,211 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:24,211 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:24,212 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:24,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:24,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:24,212 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:24,212 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:24,212 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:24,212 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:24,212 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:24,213 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:24,213 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:24,213 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:24,213 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:24,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:24,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:24,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:24,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:24,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:24,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:24,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:24,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:24,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:24,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:24,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:24,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:24,215 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:24,215 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:24,215 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:24,215 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:24,215 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:24,216 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:24,216 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:24,216 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:24,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:24,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:24,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:24,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:24,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:24,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:24,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:24,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:24,217 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:24,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:24,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:24,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:24,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:24,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:24,218 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:24,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:24,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:24,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:24,219 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:24,219 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:24,219 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:24,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:24,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:24,219 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:24,219 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:24,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:24,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:24,219 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:24,220 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:24,220 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:24,220 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:24,220 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:24,220 DEBUG [Thread-874] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:24,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:24,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:24,220 DEBUG [Thread-874] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:24,320 DEBUG [Thread-874] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:24,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:24,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:24,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:24,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:24,323 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:24,323 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:24,323 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:24,323 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:24,323 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:24,323 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:24,324 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:24,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:24,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:24,324 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:24,324 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:24,324 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:24,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:24,325 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:24,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:24,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:24,325 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:24,325 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:24,325 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:24,325 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:24,325 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:24,325 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:24,326 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:24,326 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:24,326 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:24,326 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. started... 2023-04-20 12:54:24,326 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 4782eca030b67d2c82cb20827b0bb953 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-20 12:54:24,326 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=A 2023-04-20 12:54:24,327 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:24,327 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=B 2023-04-20 12:54:24,327 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:24,327 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=C 2023-04-20 12:54:24,327 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:24,738 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042068bc0f7c919c4d61839698ba08e95a09 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042068bc0f7c919c4d61839698ba08e95a09 2023-04-20 12:54:24,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 4782eca030b67d2c82cb20827b0bb953 2023-04-20 12:54:24,884 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. as already flushing 2023-04-20 12:54:24,928 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:24,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34232 deadline: 1681995324927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:24,929 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:24,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34222 deadline: 1681995324928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:25,030 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:25,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34232 deadline: 1681995325029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:25,031 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:25,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34222 deadline: 1681995325030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:25,045 DEBUG [Thread-879] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x615d8f34 to 127.0.0.1:56673 2023-04-20 12:54:25,045 DEBUG [Thread-879] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:54:25,045 DEBUG [Thread-877] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6683e622 to 127.0.0.1:56673 2023-04-20 12:54:25,045 DEBUG [Thread-877] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:54:25,046 DEBUG [Thread-875] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x34b669b1 to 127.0.0.1:56673 2023-04-20 12:54:25,046 DEBUG [Thread-875] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:54:25,050 DEBUG [Thread-881] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4890ce53 to 127.0.0.1:56673 2023-04-20 12:54:25,050 DEBUG [Thread-881] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:54:25,144 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=329, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/A/c9592fd043924189a9af8cdd7f14703e 2023-04-20 12:54:25,233 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:25,233 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:25,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34222 deadline: 1681995325233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:25,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34232 deadline: 1681995325233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:25,536 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:25,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34222 deadline: 1681995325536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:25,537 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:25,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:34232 deadline: 1681995325537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:25,564 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=329 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/B/3c682fab4a5e4b15a8058799be94822a 2023-04-20 12:54:25,985 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=329 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/C/b43136e57d58493aa2369f6aabea5b41 2023-04-20 12:54:25,992 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/A/c9592fd043924189a9af8cdd7f14703e as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/c9592fd043924189a9af8cdd7f14703e 2023-04-20 12:54:25,999 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/c9592fd043924189a9af8cdd7f14703e, entries=150, sequenceid=329, filesize=25.4 K 2023-04-20 12:54:26,000 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/B/3c682fab4a5e4b15a8058799be94822a as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/3c682fab4a5e4b15a8058799be94822a 2023-04-20 12:54:26,006 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/3c682fab4a5e4b15a8058799be94822a, entries=150, sequenceid=329, filesize=11.9 K 2023-04-20 12:54:26,007 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/C/b43136e57d58493aa2369f6aabea5b41 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/b43136e57d58493aa2369f6aabea5b41 2023-04-20 12:54:26,012 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/b43136e57d58493aa2369f6aabea5b41, entries=150, sequenceid=329, filesize=11.9 K 2023-04-20 12:54:26,013 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 4782eca030b67d2c82cb20827b0bb953 in 1687ms, sequenceid=329, compaction requested=true 2023-04-20 12:54:26,013 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 4782eca030b67d2c82cb20827b0bb953: 2023-04-20 12:54:26,013 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:26,013 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:26,013 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:26,013 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:26,013 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:26,013 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:26,014 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:26,014 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:26,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:26,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:26,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:26,015 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:26,015 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:26,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:26,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:26,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:26,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:26,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:26,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:26,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64782240[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:26,016 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:26,016 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:26,017 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:26,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:26,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:26,017 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:26,017 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:26,017 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:26,017 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:26,017 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:26,018 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:26,018 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:26,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:26,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:26,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:26,018 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:26,018 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:26,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:26,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:26,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:26,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:26,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:26,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:26,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:26,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:26,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:26,020 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:26,020 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:26,020 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:26,020 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:26,020 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:26,021 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:26,021 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:26,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:26,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:26,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:26,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:26,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:26,021 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:26,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:26,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:26,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:26,022 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:26,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:26,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:26,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:26,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:26,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:26,023 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:26,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:26,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:26,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:26,025 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:26,025 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:26,025 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:26,025 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:26,025 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:26,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:26,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:26,025 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:26,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:26,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:26,025 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:26,026 DEBUG [Thread-874] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:26,026 DEBUG [Thread-874] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:26,026 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:26,026 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:26,026 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:26,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:26,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:26,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 4782eca030b67d2c82cb20827b0bb953 2023-04-20 12:54:26,042 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 4782eca030b67d2c82cb20827b0bb953 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-20 12:54:26,042 DEBUG [Thread-872] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5d07b2f5 to 127.0.0.1:56673 2023-04-20 12:54:26,042 DEBUG [Thread-872] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:54:26,043 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=A 2023-04-20 12:54:26,043 DEBUG [Thread-864] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6dc16927 to 127.0.0.1:56673 2023-04-20 12:54:26,043 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:54:26,044 DEBUG [Thread-864] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:54:26,044 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=B 2023-04-20 12:54:26,044 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:54:26,044 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=C 2023-04-20 12:54:26,044 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:54:26,126 DEBUG [Thread-874] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:26,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:26,468 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420e86f4eb427984476bd70270afd71adb7 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420e86f4eb427984476bd70270afd71adb7 2023-04-20 12:54:26,876 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=350, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/A/d277ee4bfd254e10a6ec13b199fbfebb 2023-04-20 12:54:27,293 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=350 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/B/d419636835e24745ac32aa9726749cb4 2023-04-20 12:54:27,713 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=350 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/C/c0e31c30c20340ed9fa24793fc787f94 2023-04-20 12:54:27,724 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/A/d277ee4bfd254e10a6ec13b199fbfebb as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/d277ee4bfd254e10a6ec13b199fbfebb 2023-04-20 12:54:27,728 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/d277ee4bfd254e10a6ec13b199fbfebb, entries=150, sequenceid=350, filesize=25.4 K 2023-04-20 12:54:27,730 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/B/d419636835e24745ac32aa9726749cb4 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/d419636835e24745ac32aa9726749cb4 2023-04-20 12:54:27,733 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/d419636835e24745ac32aa9726749cb4, entries=150, sequenceid=350, filesize=11.9 K 2023-04-20 12:54:27,734 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/C/c0e31c30c20340ed9fa24793fc787f94 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/c0e31c30c20340ed9fa24793fc787f94 2023-04-20 12:54:27,737 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/c0e31c30c20340ed9fa24793fc787f94, entries=150, sequenceid=350, filesize=11.9 K 2023-04-20 12:54:27,738 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=0 B/0 for 4782eca030b67d2c82cb20827b0bb953 in 1696ms, sequenceid=350, compaction requested=true 2023-04-20 12:54:27,738 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 4782eca030b67d2c82cb20827b0bb953: 2023-04-20 12:54:27,738 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:27,738 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-20 12:54:27,738 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-20 12:54:27,738 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:54:27,738 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:54:27,740 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 163896 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-20 12:54:27,740 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73696 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-20 12:54:27,740 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): 4782eca030b67d2c82cb20827b0bb953/A is initiating minor compaction (all files) 2023-04-20 12:54:27,740 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): 4782eca030b67d2c82cb20827b0bb953/B is initiating minor compaction (all files) 2023-04-20 12:54:27,740 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4782eca030b67d2c82cb20827b0bb953/A in TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:27,740 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4782eca030b67d2c82cb20827b0bb953/B in TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:27,740 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/90131e57d1d34aaba02b201577e6b555, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/3945b1c82cd042cfbc5275a33bd6495c, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/9fe79b8a8c184656bee9fa36fc34a207, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/7ea719d1eff746b093b63365218c91c4, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/c9592fd043924189a9af8cdd7f14703e, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/d277ee4bfd254e10a6ec13b199fbfebb] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp, totalSize=160.1 K 2023-04-20 12:54:27,740 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/47232aee7d4049408cf66cfd668cc3e0, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/9b0e00cb5aed4423969ebf15ea82d6f7, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/448b4afea6874b9fb777621d2356213e, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/ae0b2e43456148cfa1c37d7921457f3a, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/3c682fab4a5e4b15a8058799be94822a, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/d419636835e24745ac32aa9726749cb4] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp, totalSize=72.0 K 2023-04-20 12:54:27,740 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 90131e57d1d34aaba02b201577e6b555, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1681995258822 2023-04-20 12:54:27,740 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 47232aee7d4049408cf66cfd668cc3e0, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1681995258822 2023-04-20 12:54:27,741 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 3945b1c82cd042cfbc5275a33bd6495c, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=280, earliestPutTs=1681995260031 2023-04-20 12:54:27,741 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 9b0e00cb5aed4423969ebf15ea82d6f7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=280, earliestPutTs=1681995260031 2023-04-20 12:54:27,741 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 9fe79b8a8c184656bee9fa36fc34a207, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=296, earliestPutTs=1681995260957 2023-04-20 12:54:27,741 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 448b4afea6874b9fb777621d2356213e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=296, earliestPutTs=1681995260957 2023-04-20 12:54:27,741 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 7ea719d1eff746b093b63365218c91c4, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=316, earliestPutTs=1681995261627 2023-04-20 12:54:27,741 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting ae0b2e43456148cfa1c37d7921457f3a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=316, earliestPutTs=1681995261627 2023-04-20 12:54:27,742 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting c9592fd043924189a9af8cdd7f14703e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1681995262757 2023-04-20 12:54:27,742 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 3c682fab4a5e4b15a8058799be94822a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1681995262757 2023-04-20 12:54:27,742 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting d277ee4bfd254e10a6ec13b199fbfebb, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=350, earliestPutTs=1681995264921 2023-04-20 12:54:27,742 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting d419636835e24745ac32aa9726749cb4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=350, earliestPutTs=1681995264921 2023-04-20 12:54:27,752 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): 4782eca030b67d2c82cb20827b0bb953#B#compaction#176 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:27,756 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4782eca030b67d2c82cb20827b0bb953#A#compaction#175 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:28,169 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/B/8d84b93276d7449393d151bb0b430245 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/8d84b93276d7449393d151bb0b430245 2023-04-20 12:54:28,175 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 4782eca030b67d2c82cb20827b0bb953/B of 4782eca030b67d2c82cb20827b0bb953 into 8d84b93276d7449393d151bb0b430245(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:54:28,175 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4782eca030b67d2c82cb20827b0bb953: 2023-04-20 12:54:28,176 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953., storeName=4782eca030b67d2c82cb20827b0bb953/B, priority=10, startTime=1681995267738; duration=0sec 2023-04-20 12:54:28,176 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:54:28,176 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-20 12:54:28,178 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73696 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-20 12:54:28,178 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): 4782eca030b67d2c82cb20827b0bb953/C is initiating minor compaction (all files) 2023-04-20 12:54:28,178 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4782eca030b67d2c82cb20827b0bb953/C in TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:28,178 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/a4d32e8990e9476696bf214f29b269f8, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/20495ac3ed144710adae3420a75e8911, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/b612fb0fc3fb4110af33cbd5d0385ea8, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/2cd335cad36c44cabbd29aa1387990a2, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/b43136e57d58493aa2369f6aabea5b41, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/c0e31c30c20340ed9fa24793fc787f94] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp, totalSize=72.0 K 2023-04-20 12:54:28,179 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting a4d32e8990e9476696bf214f29b269f8, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1681995258822 2023-04-20 12:54:28,179 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 20495ac3ed144710adae3420a75e8911, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=280, earliestPutTs=1681995260031 2023-04-20 12:54:28,180 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting b612fb0fc3fb4110af33cbd5d0385ea8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=296, earliestPutTs=1681995260957 2023-04-20 12:54:28,180 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 2cd335cad36c44cabbd29aa1387990a2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=316, earliestPutTs=1681995261627 2023-04-20 12:54:28,180 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting b43136e57d58493aa2369f6aabea5b41, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1681995262757 2023-04-20 12:54:28,181 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting c0e31c30c20340ed9fa24793fc787f94, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=350, earliestPutTs=1681995264921 2023-04-20 12:54:28,194 DEBUG [Thread-870] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4af56759 to 127.0.0.1:56673 2023-04-20 12:54:28,194 DEBUG [Thread-870] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:54:28,198 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): 4782eca030b67d2c82cb20827b0bb953#C#compaction#177 average throughput is 1.09 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:28,205 DEBUG [Thread-868] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6322d500 to 127.0.0.1:56673 2023-04-20 12:54:28,205 DEBUG [Thread-868] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:54:28,612 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/C/2138963dbaf1474990c5913f1f098783 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/2138963dbaf1474990c5913f1f098783 2023-04-20 12:54:28,618 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 4782eca030b67d2c82cb20827b0bb953/C of 4782eca030b67d2c82cb20827b0bb953 into 2138963dbaf1474990c5913f1f098783(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:54:28,618 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4782eca030b67d2c82cb20827b0bb953: 2023-04-20 12:54:28,618 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953., storeName=4782eca030b67d2c82cb20827b0bb953/C, priority=10, startTime=1681995267738; duration=0sec 2023-04-20 12:54:28,618 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:28,993 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/A/2458818fd36c44609f59018fa0eec054 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/2458818fd36c44609f59018fa0eec054 2023-04-20 12:54:28,999 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 4782eca030b67d2c82cb20827b0bb953/A of 4782eca030b67d2c82cb20827b0bb953 into 2458818fd36c44609f59018fa0eec054(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-20 12:54:28,999 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4782eca030b67d2c82cb20827b0bb953: 2023-04-20 12:54:28,999 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953., storeName=4782eca030b67d2c82cb20827b0bb953/A, priority=10, startTime=1681995267738; duration=1sec 2023-04-20 12:54:28,999 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:33,470 DEBUG [Thread-866] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x029d68f7 to 127.0.0.1:56673 2023-04-20 12:54:33,471 DEBUG [Thread-866] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:54:33,471 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-20 12:54:33,471 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(394): wrote 96 2023-04-20 12:54:33,471 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(394): wrote 9 2023-04-20 12:54:33,471 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(394): wrote 42 2023-04-20 12:54:33,471 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(394): wrote 44 2023-04-20 12:54:33,472 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(394): wrote 88 2023-04-20 12:54:33,472 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-20 12:54:33,472 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(398): read 6045 2023-04-20 12:54:33,472 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(398): read 5848 2023-04-20 12:54:33,472 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-20 12:54:33,472 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(402): scanned 2496 2023-04-20 12:54:33,472 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(403): verified 7488 rows 2023-04-20 12:54:33,472 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(402): scanned 2526 2023-04-20 12:54:33,472 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(403): verified 7570 rows 2023-04-20 12:54:33,472 INFO [Listener at localhost.localdomain/42205] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-20 12:54:33,473 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10dd04e6 to 127.0.0.1:56673 2023-04-20 12:54:33,473 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:54:33,475 INFO [Listener at localhost.localdomain/42205] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-20 12:54:33,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-20 12:54:33,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-20 12:54:33,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-20 12:54:33,481 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681995273481"}]},"ts":"1681995273481"} 2023-04-20 12:54:33,482 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-20 12:54:33,497 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-20 12:54:33,498 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4782eca030b67d2c82cb20827b0bb953, UNASSIGN}] 2023-04-20 12:54:33,500 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=4782eca030b67d2c82cb20827b0bb953, UNASSIGN 2023-04-20 12:54:33,500 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=4782eca030b67d2c82cb20827b0bb953, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:33,501 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681995273500"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681995273500"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681995273500"}]},"ts":"1681995273500"} 2023-04-20 12:54:33,502 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure 4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947}] 2023-04-20 12:54:33,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-20 12:54:33,654 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 4782eca030b67d2c82cb20827b0bb953 2023-04-20 12:54:33,654 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 4782eca030b67d2c82cb20827b0bb953, disabling compactions & flushes 2023-04-20 12:54:33,654 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:33,655 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:33,655 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. after waiting 0 ms 2023-04-20 12:54:33,655 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:33,655 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 4782eca030b67d2c82cb20827b0bb953 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-20 12:54:33,655 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=A 2023-04-20 12:54:33,655 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:33,655 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=B 2023-04-20 12:54:33,655 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:33,655 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 4782eca030b67d2c82cb20827b0bb953, store=C 2023-04-20 12:54:33,656 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:33,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-20 12:54:34,078 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420654f6fdc383945ca950f51704bd4ab56 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420654f6fdc383945ca950f51704bd4ab56 2023-04-20 12:54:34,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-20 12:54:34,486 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=359, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/A/d7852ddfb6834712b01f7816947cf928 2023-04-20 12:54:34,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-20 12:54:34,904 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=359 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/B/decc0874492b4f0da288d49c8977f615 2023-04-20 12:54:35,132 DEBUG [regionserver/jenkins-hbase20:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-20 12:54:35,322 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=359 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/C/3c2caaef29d34302a92879fe0507c51e 2023-04-20 12:54:35,336 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/A/d7852ddfb6834712b01f7816947cf928 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/d7852ddfb6834712b01f7816947cf928 2023-04-20 12:54:35,342 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/d7852ddfb6834712b01f7816947cf928, entries=150, sequenceid=359, filesize=25.4 K 2023-04-20 12:54:35,343 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/B/decc0874492b4f0da288d49c8977f615 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/decc0874492b4f0da288d49c8977f615 2023-04-20 12:54:35,349 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/decc0874492b4f0da288d49c8977f615, entries=150, sequenceid=359, filesize=11.9 K 2023-04-20 12:54:35,350 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/.tmp/C/3c2caaef29d34302a92879fe0507c51e as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/3c2caaef29d34302a92879fe0507c51e 2023-04-20 12:54:35,357 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/3c2caaef29d34302a92879fe0507c51e, entries=150, sequenceid=359, filesize=11.9 K 2023-04-20 12:54:35,358 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 4782eca030b67d2c82cb20827b0bb953 in 1703ms, sequenceid=359, compaction requested=false 2023-04-20 12:54:35,378 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/bdea81c3a4b5466c808ffd2f2d9707d8, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/ad5a56de2a324bdf9b6b49c0035c7f8d, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/0ee6e07d36254035b35e1d2f027596c8, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/4699f61e230c48a3af60b3d01f5a6187, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/ee0511af6fa54322a01dd6f4429cf8c3, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/80a7470001324a8d9fb0a7d41e86a431, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/8ffad9d8b64f467ba18dd030b31ba473, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/953933cdec0649a6b2136ae341bfeb8c, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/3e83fc685ac543ba8f603337e9d1814f, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/9e4f8a0f5aad43af8c7a91b36d23c318, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/6aa1e2a7360f490ea4cba1362dfaedd7, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/e11e38b0331d4c2c89f977039fa63c60, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/2ab79fbd86814dd286824b3780c7ec66, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/a2016c140eaf4520946713c14959ca41, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/779ebbb96df74457ad62c5f0f409db36, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/d593e7b0d6ca435da28e77a19d1793f8, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/da93fe7491904832840b44ed958d5c90, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/90131e57d1d34aaba02b201577e6b555, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/3945b1c82cd042cfbc5275a33bd6495c, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/9fe79b8a8c184656bee9fa36fc34a207, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/7ea719d1eff746b093b63365218c91c4, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/c9592fd043924189a9af8cdd7f14703e, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/d277ee4bfd254e10a6ec13b199fbfebb] to archive 2023-04-20 12:54:35,379 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-20 12:54:35,380 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/bdea81c3a4b5466c808ffd2f2d9707d8 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/bdea81c3a4b5466c808ffd2f2d9707d8 2023-04-20 12:54:35,381 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/ad5a56de2a324bdf9b6b49c0035c7f8d to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/ad5a56de2a324bdf9b6b49c0035c7f8d 2023-04-20 12:54:35,383 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/0ee6e07d36254035b35e1d2f027596c8 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/0ee6e07d36254035b35e1d2f027596c8 2023-04-20 12:54:35,384 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/4699f61e230c48a3af60b3d01f5a6187 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/4699f61e230c48a3af60b3d01f5a6187 2023-04-20 12:54:35,385 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/ee0511af6fa54322a01dd6f4429cf8c3 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/ee0511af6fa54322a01dd6f4429cf8c3 2023-04-20 12:54:35,386 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/80a7470001324a8d9fb0a7d41e86a431 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/80a7470001324a8d9fb0a7d41e86a431 2023-04-20 12:54:35,387 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/8ffad9d8b64f467ba18dd030b31ba473 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/8ffad9d8b64f467ba18dd030b31ba473 2023-04-20 12:54:35,388 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/953933cdec0649a6b2136ae341bfeb8c to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/953933cdec0649a6b2136ae341bfeb8c 2023-04-20 12:54:35,389 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/3e83fc685ac543ba8f603337e9d1814f to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/3e83fc685ac543ba8f603337e9d1814f 2023-04-20 12:54:35,390 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/9e4f8a0f5aad43af8c7a91b36d23c318 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/9e4f8a0f5aad43af8c7a91b36d23c318 2023-04-20 12:54:35,391 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/6aa1e2a7360f490ea4cba1362dfaedd7 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/6aa1e2a7360f490ea4cba1362dfaedd7 2023-04-20 12:54:35,392 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/e11e38b0331d4c2c89f977039fa63c60 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/e11e38b0331d4c2c89f977039fa63c60 2023-04-20 12:54:35,393 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/2ab79fbd86814dd286824b3780c7ec66 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/2ab79fbd86814dd286824b3780c7ec66 2023-04-20 12:54:35,394 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/a2016c140eaf4520946713c14959ca41 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/a2016c140eaf4520946713c14959ca41 2023-04-20 12:54:35,395 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/779ebbb96df74457ad62c5f0f409db36 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/779ebbb96df74457ad62c5f0f409db36 2023-04-20 12:54:35,396 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/d593e7b0d6ca435da28e77a19d1793f8 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/d593e7b0d6ca435da28e77a19d1793f8 2023-04-20 12:54:35,397 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/da93fe7491904832840b44ed958d5c90 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/da93fe7491904832840b44ed958d5c90 2023-04-20 12:54:35,398 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/90131e57d1d34aaba02b201577e6b555 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/90131e57d1d34aaba02b201577e6b555 2023-04-20 12:54:35,399 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/3945b1c82cd042cfbc5275a33bd6495c to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/3945b1c82cd042cfbc5275a33bd6495c 2023-04-20 12:54:35,400 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/9fe79b8a8c184656bee9fa36fc34a207 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/9fe79b8a8c184656bee9fa36fc34a207 2023-04-20 12:54:35,400 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/7ea719d1eff746b093b63365218c91c4 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/7ea719d1eff746b093b63365218c91c4 2023-04-20 12:54:35,401 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/c9592fd043924189a9af8cdd7f14703e to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/c9592fd043924189a9af8cdd7f14703e 2023-04-20 12:54:35,402 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/d277ee4bfd254e10a6ec13b199fbfebb to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/d277ee4bfd254e10a6ec13b199fbfebb 2023-04-20 12:54:35,405 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/ddce1a4d850548148538945c50aa9cd0, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/a8a61105e1b243ec9eff25b470d13888, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/304fd72e2ee442599ff22e660d7efaa8, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/7c9780a9f8f842c09b2e5b444f6f33d6, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/56bf275d835a4bca911d458559450d17, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/1368f72210ee45e5bdbace57728fc31d, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/e5f1f0e6fc7f488292bd33daeaa9baea, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/102c6975cb804f4793118ac9d21d74f8, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/afa5b957f54340c49c5c58ea68ccd62a, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/013fa588f6f845858f4cd78002c7fe2c, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/282c1242f05d477e8dbc838c113c3c15, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/745e830ea18042d1b3243041d1d34066, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/158ae6e4135c404f97249c408b53540c, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/f433965d9d4e457e89750ce4e6e6094c, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/f354b6cfaae84ace8ad9250dc12e0040, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/a15b85e13a164b698b1dd9bd256aee66, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/47232aee7d4049408cf66cfd668cc3e0, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/e5bd8e8ef651491f80d07684c5206967, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/9b0e00cb5aed4423969ebf15ea82d6f7, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/448b4afea6874b9fb777621d2356213e, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/ae0b2e43456148cfa1c37d7921457f3a, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/3c682fab4a5e4b15a8058799be94822a, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/d419636835e24745ac32aa9726749cb4] to archive 2023-04-20 12:54:35,405 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-20 12:54:35,406 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/ddce1a4d850548148538945c50aa9cd0 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/ddce1a4d850548148538945c50aa9cd0 2023-04-20 12:54:35,407 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/a8a61105e1b243ec9eff25b470d13888 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/a8a61105e1b243ec9eff25b470d13888 2023-04-20 12:54:35,408 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/304fd72e2ee442599ff22e660d7efaa8 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/304fd72e2ee442599ff22e660d7efaa8 2023-04-20 12:54:35,409 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/7c9780a9f8f842c09b2e5b444f6f33d6 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/7c9780a9f8f842c09b2e5b444f6f33d6 2023-04-20 12:54:35,410 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/56bf275d835a4bca911d458559450d17 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/56bf275d835a4bca911d458559450d17 2023-04-20 12:54:35,411 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/1368f72210ee45e5bdbace57728fc31d to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/1368f72210ee45e5bdbace57728fc31d 2023-04-20 12:54:35,412 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/e5f1f0e6fc7f488292bd33daeaa9baea to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/e5f1f0e6fc7f488292bd33daeaa9baea 2023-04-20 12:54:35,413 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/102c6975cb804f4793118ac9d21d74f8 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/102c6975cb804f4793118ac9d21d74f8 2023-04-20 12:54:35,414 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/afa5b957f54340c49c5c58ea68ccd62a to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/afa5b957f54340c49c5c58ea68ccd62a 2023-04-20 12:54:35,415 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/013fa588f6f845858f4cd78002c7fe2c to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/013fa588f6f845858f4cd78002c7fe2c 2023-04-20 12:54:35,415 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/282c1242f05d477e8dbc838c113c3c15 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/282c1242f05d477e8dbc838c113c3c15 2023-04-20 12:54:35,416 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/745e830ea18042d1b3243041d1d34066 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/745e830ea18042d1b3243041d1d34066 2023-04-20 12:54:35,417 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/158ae6e4135c404f97249c408b53540c to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/158ae6e4135c404f97249c408b53540c 2023-04-20 12:54:35,418 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/f433965d9d4e457e89750ce4e6e6094c to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/f433965d9d4e457e89750ce4e6e6094c 2023-04-20 12:54:35,419 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/f354b6cfaae84ace8ad9250dc12e0040 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/f354b6cfaae84ace8ad9250dc12e0040 2023-04-20 12:54:35,420 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/a15b85e13a164b698b1dd9bd256aee66 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/a15b85e13a164b698b1dd9bd256aee66 2023-04-20 12:54:35,420 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/47232aee7d4049408cf66cfd668cc3e0 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/47232aee7d4049408cf66cfd668cc3e0 2023-04-20 12:54:35,421 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/e5bd8e8ef651491f80d07684c5206967 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/e5bd8e8ef651491f80d07684c5206967 2023-04-20 12:54:35,422 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/9b0e00cb5aed4423969ebf15ea82d6f7 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/9b0e00cb5aed4423969ebf15ea82d6f7 2023-04-20 12:54:35,423 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/448b4afea6874b9fb777621d2356213e to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/448b4afea6874b9fb777621d2356213e 2023-04-20 12:54:35,424 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/ae0b2e43456148cfa1c37d7921457f3a to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/ae0b2e43456148cfa1c37d7921457f3a 2023-04-20 12:54:35,424 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/3c682fab4a5e4b15a8058799be94822a to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/3c682fab4a5e4b15a8058799be94822a 2023-04-20 12:54:35,425 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/d419636835e24745ac32aa9726749cb4 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/d419636835e24745ac32aa9726749cb4 2023-04-20 12:54:35,428 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/512e907748d84b4dbe5ad77ddc0f8db3, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/302b05d598d343c4a78fc78c11bd70e5, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/99970ffb2c2d4b059afff7e67052475f, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/4c5eb47fd0fc45b592c4095368ef7946, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/41035605d1784bc38301d3ab4a68d6f3, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/bb39d0be8dfe41ae9cebe9cdd7286a09, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/9576fe8fe1544d8cb52b845b37be9110, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/08bde1af4d224a2ab8896a97fbc960be, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/ed52df96adb34631a4e6b445e7d2d79b, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/f227800671794e118b2b0c2a48bf91c2, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/e2e71db2727745a2abf4a9d38bfa9ef3, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/19208133e2bb420eaff8f660750f199e, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/a86610b1e68849dc8a72f18d440c7eb0, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/1efa56be7ea146a1a80af03065044ec1, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/40c1ba9d1f484f9693bcf44bf1aafb67, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/f0e701d1368147e694a623ec38f3bee7, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/a4d32e8990e9476696bf214f29b269f8, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/9d91e996788048a4a58153e762996a3f, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/20495ac3ed144710adae3420a75e8911, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/b612fb0fc3fb4110af33cbd5d0385ea8, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/2cd335cad36c44cabbd29aa1387990a2, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/b43136e57d58493aa2369f6aabea5b41, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/c0e31c30c20340ed9fa24793fc787f94] to archive 2023-04-20 12:54:35,428 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-20 12:54:35,430 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/512e907748d84b4dbe5ad77ddc0f8db3 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/512e907748d84b4dbe5ad77ddc0f8db3 2023-04-20 12:54:35,431 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/302b05d598d343c4a78fc78c11bd70e5 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/302b05d598d343c4a78fc78c11bd70e5 2023-04-20 12:54:35,432 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/99970ffb2c2d4b059afff7e67052475f to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/99970ffb2c2d4b059afff7e67052475f 2023-04-20 12:54:35,432 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/4c5eb47fd0fc45b592c4095368ef7946 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/4c5eb47fd0fc45b592c4095368ef7946 2023-04-20 12:54:35,433 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/41035605d1784bc38301d3ab4a68d6f3 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/41035605d1784bc38301d3ab4a68d6f3 2023-04-20 12:54:35,434 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/bb39d0be8dfe41ae9cebe9cdd7286a09 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/bb39d0be8dfe41ae9cebe9cdd7286a09 2023-04-20 12:54:35,435 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/9576fe8fe1544d8cb52b845b37be9110 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/9576fe8fe1544d8cb52b845b37be9110 2023-04-20 12:54:35,436 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/08bde1af4d224a2ab8896a97fbc960be to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/08bde1af4d224a2ab8896a97fbc960be 2023-04-20 12:54:35,437 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/ed52df96adb34631a4e6b445e7d2d79b to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/ed52df96adb34631a4e6b445e7d2d79b 2023-04-20 12:54:35,437 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/f227800671794e118b2b0c2a48bf91c2 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/f227800671794e118b2b0c2a48bf91c2 2023-04-20 12:54:35,438 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/e2e71db2727745a2abf4a9d38bfa9ef3 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/e2e71db2727745a2abf4a9d38bfa9ef3 2023-04-20 12:54:35,439 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/19208133e2bb420eaff8f660750f199e to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/19208133e2bb420eaff8f660750f199e 2023-04-20 12:54:35,440 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/a86610b1e68849dc8a72f18d440c7eb0 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/a86610b1e68849dc8a72f18d440c7eb0 2023-04-20 12:54:35,441 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/1efa56be7ea146a1a80af03065044ec1 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/1efa56be7ea146a1a80af03065044ec1 2023-04-20 12:54:35,441 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/40c1ba9d1f484f9693bcf44bf1aafb67 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/40c1ba9d1f484f9693bcf44bf1aafb67 2023-04-20 12:54:35,442 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/f0e701d1368147e694a623ec38f3bee7 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/f0e701d1368147e694a623ec38f3bee7 2023-04-20 12:54:35,443 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/a4d32e8990e9476696bf214f29b269f8 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/a4d32e8990e9476696bf214f29b269f8 2023-04-20 12:54:35,444 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/9d91e996788048a4a58153e762996a3f to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/9d91e996788048a4a58153e762996a3f 2023-04-20 12:54:35,445 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/20495ac3ed144710adae3420a75e8911 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/20495ac3ed144710adae3420a75e8911 2023-04-20 12:54:35,446 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/b612fb0fc3fb4110af33cbd5d0385ea8 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/b612fb0fc3fb4110af33cbd5d0385ea8 2023-04-20 12:54:35,447 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/2cd335cad36c44cabbd29aa1387990a2 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/2cd335cad36c44cabbd29aa1387990a2 2023-04-20 12:54:35,448 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/b43136e57d58493aa2369f6aabea5b41 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/b43136e57d58493aa2369f6aabea5b41 2023-04-20 12:54:35,449 DEBUG [StoreCloser-TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/c0e31c30c20340ed9fa24793fc787f94 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/c0e31c30c20340ed9fa24793fc787f94 2023-04-20 12:54:35,453 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/recovered.edits/362.seqid, newMaxSeqId=362, maxSeqId=4 2023-04-20 12:54:35,455 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953. 2023-04-20 12:54:35,455 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 4782eca030b67d2c82cb20827b0bb953: 2023-04-20 12:54:35,456 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 4782eca030b67d2c82cb20827b0bb953 2023-04-20 12:54:35,456 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=4782eca030b67d2c82cb20827b0bb953, regionState=CLOSED 2023-04-20 12:54:35,457 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681995275456"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681995275456"}]},"ts":"1681995275456"} 2023-04-20 12:54:35,459 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-04-20 12:54:35,459 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure 4782eca030b67d2c82cb20827b0bb953, server=jenkins-hbase20.apache.org,36277,1681995213947 in 1.9550 sec 2023-04-20 12:54:35,461 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-04-20 12:54:35,461 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=4782eca030b67d2c82cb20827b0bb953, UNASSIGN in 1.9610 sec 2023-04-20 12:54:35,461 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681995275461"}]},"ts":"1681995275461"} 2023-04-20 12:54:35,462 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-20 12:54:35,464 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-20 12:54:35,466 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9890 sec 2023-04-20 12:54:35,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-20 12:54:35,588 INFO [Listener at localhost.localdomain/42205] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-04-20 12:54:35,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-20 12:54:35,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 12:54:35,593 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 12:54:35,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-20 12:54:35,594 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 12:54:35,602 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953 2023-04-20 12:54:35,606 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A, FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B, FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C, FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/recovered.edits] 2023-04-20 12:54:35,610 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/2458818fd36c44609f59018fa0eec054 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/2458818fd36c44609f59018fa0eec054 2023-04-20 12:54:35,612 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/d7852ddfb6834712b01f7816947cf928 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/A/d7852ddfb6834712b01f7816947cf928 2023-04-20 12:54:35,615 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/8d84b93276d7449393d151bb0b430245 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/8d84b93276d7449393d151bb0b430245 2023-04-20 12:54:35,616 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/decc0874492b4f0da288d49c8977f615 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/B/decc0874492b4f0da288d49c8977f615 2023-04-20 12:54:35,619 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/2138963dbaf1474990c5913f1f098783 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/2138963dbaf1474990c5913f1f098783 2023-04-20 12:54:35,621 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/3c2caaef29d34302a92879fe0507c51e to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/C/3c2caaef29d34302a92879fe0507c51e 2023-04-20 12:54:35,625 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/recovered.edits/362.seqid to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953/recovered.edits/362.seqid 2023-04-20 12:54:35,626 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/4782eca030b67d2c82cb20827b0bb953 2023-04-20 12:54:35,626 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-20 12:54:35,627 DEBUG [PEWorker-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-20 12:54:35,628 DEBUG [PEWorker-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-20 12:54:35,633 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420073c5bf151ba4ab0a386a6ac0b6c39cd to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420073c5bf151ba4ab0a386a6ac0b6c39cd 2023-04-20 12:54:35,635 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304201496a3190554484299449d691e085eb7 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304201496a3190554484299449d691e085eb7 2023-04-20 12:54:35,638 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042016df5e1c35c147218d0637d743709620 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042016df5e1c35c147218d0637d743709620 2023-04-20 12:54:35,640 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304201ae1f2e7a5564484ab1a9e0f48ab07bf to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304201ae1f2e7a5564484ab1a9e0f48ab07bf 2023-04-20 12:54:35,642 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304203ba05751ba7440678e88e88a7606ce83 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304203ba05751ba7440678e88e88a7606ce83 2023-04-20 12:54:35,643 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420477cdddc064f4877a982ffe62083577e to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420477cdddc064f4877a982ffe62083577e 2023-04-20 12:54:35,644 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304204a82725e5e834e2fba9536d1b37bfc9b to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304204a82725e5e834e2fba9536d1b37bfc9b 2023-04-20 12:54:35,645 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042050a71cfcfc3a4cec9c98add7c31db99e to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042050a71cfcfc3a4cec9c98add7c31db99e 2023-04-20 12:54:35,646 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420511563ada8054c5b9c7cf23c56f3923d to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420511563ada8054c5b9c7cf23c56f3923d 2023-04-20 12:54:35,647 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420654f6fdc383945ca950f51704bd4ab56 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420654f6fdc383945ca950f51704bd4ab56 2023-04-20 12:54:35,648 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042068bc0f7c919c4d61839698ba08e95a09 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042068bc0f7c919c4d61839698ba08e95a09 2023-04-20 12:54:35,649 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304209de7a642dda54b7bacc027da8336d1f4 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304209de7a642dda54b7bacc027da8336d1f4 2023-04-20 12:54:35,650 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304209ec1662b1036427c8f912a8edaf1d08f to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304209ec1662b1036427c8f912a8edaf1d08f 2023-04-20 12:54:35,651 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420a3f445728b8f406d8323fba90e08bb14 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420a3f445728b8f406d8323fba90e08bb14 2023-04-20 12:54:35,652 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420b96848c3f1ed478e9422ba592db513dc to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420b96848c3f1ed478e9422ba592db513dc 2023-04-20 12:54:35,653 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420c7c681349f614e18ac77c7270dc27182 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420c7c681349f614e18ac77c7270dc27182 2023-04-20 12:54:35,654 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420d39b9704507e4e15ae585c62fc57b041 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420d39b9704507e4e15ae585c62fc57b041 2023-04-20 12:54:35,655 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420dc67d52fb94341c9a047218455278eec to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420dc67d52fb94341c9a047218455278eec 2023-04-20 12:54:35,656 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420e86f4eb427984476bd70270afd71adb7 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420e86f4eb427984476bd70270afd71adb7 2023-04-20 12:54:35,656 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420fcc1275c57894401ac37098bf162bea4 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420fcc1275c57894401ac37098bf162bea4 2023-04-20 12:54:35,657 DEBUG [PEWorker-2] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-20 12:54:35,659 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 12:54:35,660 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-20 12:54:35,662 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-20 12:54:35,663 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 12:54:35,663 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-20 12:54:35,663 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681995275663"}]},"ts":"9223372036854775807"} 2023-04-20 12:54:35,664 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-20 12:54:35,664 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 4782eca030b67d2c82cb20827b0bb953, NAME => 'TestAcidGuarantees,,1681995241853.4782eca030b67d2c82cb20827b0bb953.', STARTKEY => '', ENDKEY => ''}] 2023-04-20 12:54:35,664 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-20 12:54:35,664 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681995275664"}]},"ts":"9223372036854775807"} 2023-04-20 12:54:35,666 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-20 12:54:35,667 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 12:54:35,668 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 78 msec 2023-04-20 12:54:35,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-20 12:54:35,696 INFO [Listener at localhost.localdomain/42205] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-04-20 12:54:35,708 INFO [Listener at localhost.localdomain/42205] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=229 (was 236), OpenFileDescriptor=523 (was 559), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=322 (was 344), ProcessCount=170 (was 172), AvailableMemoryMB=9231 (was 9309) 2023-04-20 12:54:35,721 INFO [Listener at localhost.localdomain/42205] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=229, OpenFileDescriptor=523, MaxFileDescriptor=60000, SystemLoadAverage=322, ProcessCount=170, AvailableMemoryMB=9230 2023-04-20 12:54:35,724 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-20 12:54:35,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-20 12:54:35,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-20 12:54:35,727 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-20 12:54:35,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-04-20 12:54:35,728 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-20 12:54:35,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-20 12:54:35,729 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551 2023-04-20 12:54:35,729 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551 empty. 2023-04-20 12:54:35,730 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551 2023-04-20 12:54:35,730 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-20 12:54:35,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-20 12:54:36,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-20 12:54:36,142 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-20 12:54:36,147 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 969c768384097479d79029eb29913551, NAME => 'TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp 2023-04-20 12:54:36,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-20 12:54:36,561 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-20 12:54:36,562 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 969c768384097479d79029eb29913551, disabling compactions & flushes 2023-04-20 12:54:36,562 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:36,562 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:36,562 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. after waiting 0 ms 2023-04-20 12:54:36,562 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:36,562 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:36,563 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:36,569 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-20 12:54:36,570 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681995276570"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681995276570"}]},"ts":"1681995276570"} 2023-04-20 12:54:36,572 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-20 12:54:36,573 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-20 12:54:36,573 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681995276573"}]},"ts":"1681995276573"} 2023-04-20 12:54:36,574 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-20 12:54:36,578 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=969c768384097479d79029eb29913551, ASSIGN}] 2023-04-20 12:54:36,580 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=969c768384097479d79029eb29913551, ASSIGN 2023-04-20 12:54:36,581 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=969c768384097479d79029eb29913551, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,36277,1681995213947; forceNewPlan=false, retain=false 2023-04-20 12:54:36,733 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=969c768384097479d79029eb29913551, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:36,734 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681995276733"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681995276733"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681995276733"}]},"ts":"1681995276733"} 2023-04-20 12:54:36,736 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure 969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947}] 2023-04-20 12:54:36,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-20 12:54:36,901 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:36,901 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 969c768384097479d79029eb29913551, NAME => 'TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.', STARTKEY => '', ENDKEY => ''} 2023-04-20 12:54:36,901 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 969c768384097479d79029eb29913551 2023-04-20 12:54:36,901 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-20 12:54:36,901 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 969c768384097479d79029eb29913551 2023-04-20 12:54:36,901 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 969c768384097479d79029eb29913551 2023-04-20 12:54:36,903 INFO [StoreOpener-969c768384097479d79029eb29913551-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 969c768384097479d79029eb29913551 2023-04-20 12:54:36,905 DEBUG [StoreOpener-969c768384097479d79029eb29913551-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A 2023-04-20 12:54:36,905 DEBUG [StoreOpener-969c768384097479d79029eb29913551-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A 2023-04-20 12:54:36,906 INFO [StoreOpener-969c768384097479d79029eb29913551-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 12:54:36,906 INFO [StoreOpener-969c768384097479d79029eb29913551-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 969c768384097479d79029eb29913551 columnFamilyName A 2023-04-20 12:54:36,907 INFO [StoreOpener-969c768384097479d79029eb29913551-1] regionserver.HStore(310): Store=969c768384097479d79029eb29913551/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 12:54:36,907 INFO [StoreOpener-969c768384097479d79029eb29913551-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 969c768384097479d79029eb29913551 2023-04-20 12:54:36,909 DEBUG [StoreOpener-969c768384097479d79029eb29913551-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B 2023-04-20 12:54:36,909 DEBUG [StoreOpener-969c768384097479d79029eb29913551-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B 2023-04-20 12:54:36,909 INFO [StoreOpener-969c768384097479d79029eb29913551-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 12:54:36,909 INFO [StoreOpener-969c768384097479d79029eb29913551-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 969c768384097479d79029eb29913551 columnFamilyName B 2023-04-20 12:54:36,910 INFO [StoreOpener-969c768384097479d79029eb29913551-1] regionserver.HStore(310): Store=969c768384097479d79029eb29913551/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 12:54:36,910 INFO [StoreOpener-969c768384097479d79029eb29913551-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 969c768384097479d79029eb29913551 2023-04-20 12:54:36,911 DEBUG [StoreOpener-969c768384097479d79029eb29913551-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C 2023-04-20 12:54:36,911 DEBUG [StoreOpener-969c768384097479d79029eb29913551-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C 2023-04-20 12:54:36,912 INFO [StoreOpener-969c768384097479d79029eb29913551-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 12:54:36,912 INFO [StoreOpener-969c768384097479d79029eb29913551-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 969c768384097479d79029eb29913551 columnFamilyName C 2023-04-20 12:54:36,912 INFO [StoreOpener-969c768384097479d79029eb29913551-1] regionserver.HStore(310): Store=969c768384097479d79029eb29913551/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 12:54:36,912 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:36,913 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551 2023-04-20 12:54:36,914 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551 2023-04-20 12:54:36,917 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-20 12:54:36,919 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 969c768384097479d79029eb29913551 2023-04-20 12:54:36,921 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-20 12:54:36,921 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 969c768384097479d79029eb29913551; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=58990710, jitterRate=-0.12096992135047913}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-20 12:54:36,921 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:36,922 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551., pid=30, masterSystemTime=1681995276890 2023-04-20 12:54:36,923 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:36,923 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:36,924 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=969c768384097479d79029eb29913551, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:36,924 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681995276923"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1681995276923"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681995276923"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681995276923"}]},"ts":"1681995276923"} 2023-04-20 12:54:36,926 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-04-20 12:54:36,926 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure 969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 in 189 msec 2023-04-20 12:54:36,927 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-04-20 12:54:36,927 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=969c768384097479d79029eb29913551, ASSIGN in 348 msec 2023-04-20 12:54:36,927 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-20 12:54:36,928 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681995276928"}]},"ts":"1681995276928"} 2023-04-20 12:54:36,928 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-20 12:54:36,930 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-20 12:54:36,931 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.2060 sec 2023-04-20 12:54:37,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-20 12:54:37,840 INFO [Listener at localhost.localdomain/42205] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-04-20 12:54:37,842 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x73ef029c to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:54:37,849 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@f210b07, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:54:37,851 DEBUG [Listener at localhost.localdomain/42205] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:54:37,853 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40536, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:54:37,855 DEBUG [Listener at localhost.localdomain/42205] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-20 12:54:37,858 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:53190, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-20 12:54:37,860 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x2135e03b to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:54:37,864 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2a45fd63, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:54:37,865 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x5f2b45f8 to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:54:37,868 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5191421e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:54:37,869 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x24e1e878 to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:54:37,871 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@21e07d99, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:54:37,872 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x7e9921a7 to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:54:37,876 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5590a424, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:54:37,876 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x5ffcf146 to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:54:37,881 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@68df7707, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:54:37,881 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x07742ca6 to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:54:37,891 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@14001400, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:54:37,891 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x353e11a8 to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:54:37,899 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@13091ab5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:54:37,899 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x12f583c6 to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:54:37,907 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@57836361, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:54:37,907 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x6635f17a to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:54:37,915 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5860b5c9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:54:37,915 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x38204078 to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:54:37,924 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@71d7d827, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:54:37,930 DEBUG [hconnection-0x249b088-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:54:37,930 DEBUG [hconnection-0x177ddb04-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:54:37,931 DEBUG [hconnection-0x62ab7ee-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:54:37,932 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40560, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:54:37,934 DEBUG [hconnection-0x22e0718f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:54:37,934 DEBUG [hconnection-0x56532e8a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:54:37,935 DEBUG [hconnection-0x3ddc38dd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:54:37,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:37,939 DEBUG [hconnection-0xf1eb63d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:54:37,939 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40544, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:54:37,939 DEBUG [hconnection-0x70fa5e8c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:54:37,939 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40554, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:54:37,939 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40568, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:54:37,940 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40570, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:54:37,940 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40588, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:54:37,940 DEBUG [hconnection-0x4cabe648-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:54:37,938 DEBUG [hconnection-0x1467ebd5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:54:37,943 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40606, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:54:37,944 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40598, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:54:37,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:37,944 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40616, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:54:37,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:37,945 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:37,945 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:37,945 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:37,945 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:37,946 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:37,946 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:37,947 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:37,947 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:37,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:37,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:37,947 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:37,947 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:37,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:37,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:37,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:37,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:37,948 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:37,948 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:37,949 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40626, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:54:37,949 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:37,950 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:37,950 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:37,950 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:37,950 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:37,951 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:37,951 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:37,952 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:37,954 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 969c768384097479d79029eb29913551 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-20 12:54:37,957 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=A 2023-04-20 12:54:37,957 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:37,957 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=B 2023-04-20 12:54:37,957 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:37,957 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=C 2023-04-20 12:54:37,957 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:37,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 969c768384097479d79029eb29913551 2023-04-20 12:54:37,962 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:37,979 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681995277963, max timestamp=1681995277976Num uniques 859; 2023-04-20 12:54:37,979 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681995277963, max timestamp=1681995277976Num uniques 859; 2023-04-20 12:54:37,979 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-20 12:54:37,979 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681995277963, max timestamp=1681995277976Num uniques 859; 2023-04-20 12:54:37,979 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-20 12:54:37,979 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-20 12:54:37,994 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:37,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40570 deadline: 1681995337991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:37,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681995277987, max timestamp=1681995277993Num uniques 1168; 2023-04-20 12:54:37,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=53488, new segment data size=9978, suffix heap size=149440,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1168, new segment cells count=218 2023-04-20 12:54:37,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681995277987, max timestamp=1681995277993Num uniques 1168; 2023-04-20 12:54:37,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=53488, new segment data size=9978, suffix heap size=149440,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1168, new segment cells count=218 2023-04-20 12:54:37,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681995277987, max timestamp=1681995277993Num uniques 1168; 2023-04-20 12:54:37,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=53488, new segment data size=9978, suffix heap size=149440,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1168, new segment cells count=218 2023-04-20 12:54:38,016 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:38,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40568 deadline: 1681995338013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,017 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:38,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40616 deadline: 1681995338014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681995278006, max timestamp=1681995278015Num uniques 1077; 2023-04-20 12:54:38,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-20 12:54:38,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681995278006, max timestamp=1681995278015Num uniques 1077; 2023-04-20 12:54:38,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-20 12:54:38,020 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681995278006, max timestamp=1681995278015Num uniques 1077; 2023-04-20 12:54:38,020 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-20 12:54:38,048 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681995278039, max timestamp=1681995278045Num uniques 1086; 2023-04-20 12:54:38,048 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681995278039, max timestamp=1681995278045Num uniques 1036; 2023-04-20 12:54:38,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-04-20 12:54:38,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681995278039, max timestamp=1681995278045Num uniques 1036; 2023-04-20 12:54:38,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-20 12:54:38,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-20 12:54:38,083 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:38,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40560 deadline: 1681995338081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681995278076, max timestamp=1681995278082Num uniques 1095; 2023-04-20 12:54:38,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-20 12:54:38,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681995278076, max timestamp=1681995278082Num uniques 1095; 2023-04-20 12:54:38,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-20 12:54:38,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681995278076, max timestamp=1681995278082Num uniques 1095; 2023-04-20 12:54:38,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-20 12:54:38,085 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:38,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40554 deadline: 1681995338084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,133 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681995278121, max timestamp=1681995278131Num uniques 1054; 2023-04-20 12:54:38,133 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=11630, suffix heap size=137584,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=254 2023-04-20 12:54:38,133 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681995278121, max timestamp=1681995278131Num uniques 1054; 2023-04-20 12:54:38,133 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681995278121, max timestamp=1681995278131Num uniques 1054; 2023-04-20 12:54:38,133 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=11630, suffix heap size=137584,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=254 2023-04-20 12:54:38,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=11630, suffix heap size=137584,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=254 2023-04-20 12:54:38,150 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:38,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40568 deadline: 1681995338148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,150 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:38,150 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:38,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40570 deadline: 1681995338149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40616 deadline: 1681995338149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,187 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:38,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40560 deadline: 1681995338186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,191 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:38,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40554 deadline: 1681995338191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,252 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:38,252 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:38,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40568 deadline: 1681995338251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40616 deadline: 1681995338251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,252 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:38,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40570 deadline: 1681995338251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,376 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/2ec5f1d068c749fab513bba61c5aaa9b 2023-04-20 12:54:38,389 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/41beeff62f1c4175be854898a83d1cbd 2023-04-20 12:54:38,389 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:38,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40560 deadline: 1681995338388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,394 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:38,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40554 deadline: 1681995338393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,401 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/56045c8f78ab4a9aa1fc117ae886e1e8 2023-04-20 12:54:38,410 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/2ec5f1d068c749fab513bba61c5aaa9b as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/2ec5f1d068c749fab513bba61c5aaa9b 2023-04-20 12:54:38,420 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/2ec5f1d068c749fab513bba61c5aaa9b, entries=150, sequenceid=9, filesize=11.6 K 2023-04-20 12:54:38,421 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/41beeff62f1c4175be854898a83d1cbd as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/41beeff62f1c4175be854898a83d1cbd 2023-04-20 12:54:38,426 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/41beeff62f1c4175be854898a83d1cbd, entries=150, sequenceid=9, filesize=11.6 K 2023-04-20 12:54:38,427 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/56045c8f78ab4a9aa1fc117ae886e1e8 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/56045c8f78ab4a9aa1fc117ae886e1e8 2023-04-20 12:54:38,432 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/56045c8f78ab4a9aa1fc117ae886e1e8, entries=150, sequenceid=9, filesize=11.6 K 2023-04-20 12:54:38,432 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for 969c768384097479d79029eb29913551 in 479ms, sequenceid=9, compaction requested=false 2023-04-20 12:54:38,432 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-20 12:54:38,434 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:38,434 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:38,434 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:38,434 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:38,434 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,434 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:38,434 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:38,436 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,436 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:38,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:38,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:38,436 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:38,436 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:38,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:38,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:38,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:38,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:38,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:38,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8990393[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:38,438 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:38,438 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:38,438 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:38,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:38,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:38,439 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:38,439 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:38,439 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:38,439 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,439 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:38,440 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,440 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:38,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:38,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:38,440 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:38,440 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:38,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:38,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:38,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:38,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:38,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:38,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:38,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:38,443 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:38,443 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:38,443 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:38,443 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:38,443 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:38,444 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:38,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:38,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:38,444 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:38,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:38,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:38,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:38,445 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:38,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:38,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:38,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:38,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:38,446 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:38,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:38,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:38,447 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:38,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:38,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 969c768384097479d79029eb29913551 2023-04-20 12:54:38,455 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 969c768384097479d79029eb29913551 3/3 column families, dataSize=154.31 KB heapSize=436.55 KB 2023-04-20 12:54:38,455 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,456 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:38,456 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:38,455 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:38,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:38,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:38,456 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:38,456 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:38,456 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,456 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:38,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:38,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:38,456 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:38,456 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=A 2023-04-20 12:54:38,456 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:38,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681995278135, max timestamp=1681995278454Num uniques 1113; 2023-04-20 12:54:38,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681995278135, max timestamp=1681995278454Num uniques 1113; 2023-04-20 12:54:38,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=7458, suffix heap size=143720,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=163 2023-04-20 12:54:38,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:38,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:38,456 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:38,457 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:38,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=7458, suffix heap size=143720,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=163 2023-04-20 12:54:38,456 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:38,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681995278135, max timestamp=1681995278454Num uniques 1113; 2023-04-20 12:54:38,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=7458, suffix heap size=143720,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=163 2023-04-20 12:54:38,457 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:54:38,457 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=B 2023-04-20 12:54:38,457 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:54:38,457 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=C 2023-04-20 12:54:38,457 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:54:38,464 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=135 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/ee09fad504084976853fc119bbbebf14 2023-04-20 12:54:38,477 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=135 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/7d4e295258d34a15a1d9ff50d2b65ad4 2023-04-20 12:54:38,479 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681995278469, max timestamp=1681995278477Num uniques 859; 2023-04-20 12:54:38,479 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681995278469, max timestamp=1681995278477Num uniques 859; 2023-04-20 12:54:38,479 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 12:54:38,479 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 12:54:38,479 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681995278469, max timestamp=1681995278477Num uniques 859; 2023-04-20 12:54:38,479 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 12:54:38,491 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=135 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/9d3c20565923423fbc69c1099c659957 2023-04-20 12:54:38,494 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/ee09fad504084976853fc119bbbebf14 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/ee09fad504084976853fc119bbbebf14 2023-04-20 12:54:38,498 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/ee09fad504084976853fc119bbbebf14, entries=150, sequenceid=135, filesize=11.8 K 2023-04-20 12:54:38,498 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/7d4e295258d34a15a1d9ff50d2b65ad4 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/7d4e295258d34a15a1d9ff50d2b65ad4 2023-04-20 12:54:38,501 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/7d4e295258d34a15a1d9ff50d2b65ad4, entries=150, sequenceid=135, filesize=11.8 K 2023-04-20 12:54:38,502 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/9d3c20565923423fbc69c1099c659957 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/9d3c20565923423fbc69c1099c659957 2023-04-20 12:54:38,505 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/9d3c20565923423fbc69c1099c659957, entries=150, sequenceid=135, filesize=11.8 K 2023-04-20 12:54:38,506 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~105.66 KB/108192, currentSize=53.67 KB/54960 for 969c768384097479d79029eb29913551 in 51ms, sequenceid=135, compaction requested=false 2023-04-20 12:54:38,507 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:38,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 969c768384097479d79029eb29913551 2023-04-20 12:54:38,507 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 969c768384097479d79029eb29913551 3/3 column families, dataSize=73.80 KB heapSize=201 KB 2023-04-20 12:54:38,507 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=A 2023-04-20 12:54:38,507 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:54:38,507 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=B 2023-04-20 12:54:38,507 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:54:38,507 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=C 2023-04-20 12:54:38,507 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:54:38,533 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:38,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40616 deadline: 1681995338530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,535 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:38,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40568 deadline: 1681995338531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,535 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:38,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40570 deadline: 1681995338532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,557 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:38,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:38,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:38,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:38,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:38,560 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:38,560 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:38,560 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:38,560 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:38,560 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:38,560 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:38,561 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,561 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:38,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:38,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:38,561 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,561 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:38,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:38,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:38,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:38,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:38,562 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:38,562 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:38,562 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:38,562 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:38,562 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:38,563 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:38,563 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:38,563 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:38,563 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:38,563 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:38,563 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:38,563 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:38,563 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:38,563 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:38,563 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,563 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:38,563 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:38,564 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,564 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:38,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:38,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:38,564 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:38,564 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:38,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:38,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:38,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:38,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:38,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:38,565 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:38,565 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:38,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57430ff[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:38,566 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:38,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:38,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:38,567 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:38,567 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,567 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:38,567 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:38,567 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:38,567 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,567 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:38,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:38,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:38,567 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:38,568 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:38,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:38,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:38,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:38,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:38,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:38,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:38,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:38,569 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:38,569 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:38,569 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:38,569 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:38,569 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:38,570 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:38,570 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:38,570 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:38,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:38,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:38,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:38,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:38,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:38,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:38,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:38,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:38,571 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:38,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:38,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:38,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:38,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,572 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:38,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:38,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,573 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,573 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:38,573 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:38,573 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:38,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:38,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:38,573 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:38,574 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:38,574 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:38,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:38,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:38,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:38,574 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:38,574 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:38,574 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:38,574 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:38,574 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:38,637 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:38,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40616 deadline: 1681995338634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,637 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:38,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40568 deadline: 1681995338636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,638 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:38,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40570 deadline: 1681995338636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,674 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:38,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:38,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:38,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:38,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:38,677 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:38,677 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:38,677 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:38,677 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:38,678 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:38,678 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:38,678 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:38,678 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:38,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:38,679 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,679 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:38,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:38,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:38,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:38,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:38,679 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:38,679 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:38,679 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:38,679 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:38,679 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:38,679 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:38,680 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:38,680 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:38,680 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:38,680 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:38,680 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:38,680 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:38,680 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:38,680 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,680 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:38,680 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:38,680 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,680 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:38,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:38,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:38,681 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:38,681 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:38,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:38,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:38,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:38,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:38,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:38,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@301e1a1f[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:38,682 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:38,682 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:38,683 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:38,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:38,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:38,683 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:38,683 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:38,683 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:38,683 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,683 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:38,684 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,684 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:38,684 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:38,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:38,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:38,684 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:38,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:38,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:38,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:38,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:38,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:38,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:38,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:38,686 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:38,686 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:38,686 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:38,686 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:38,686 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:38,687 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:38,687 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:38,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:38,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:38,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:38,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:38,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:38,687 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:38,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:38,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:38,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:38,687 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:38,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:38,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:38,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:38,689 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:38,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:38,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,690 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,690 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:38,690 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:38,690 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,690 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:38,690 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:38,690 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:38,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:38,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:38,690 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:38,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:38,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:38,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:38,690 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:38,691 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:38,691 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:38,691 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:38,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:38,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:38,693 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:38,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40560 deadline: 1681995338691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,699 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:38,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40554 deadline: 1681995338697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,791 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:38,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:38,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:38,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:38,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:38,796 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:38,796 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:38,796 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:38,796 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:38,797 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:38,797 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:38,798 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,798 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:38,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:38,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:38,798 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,798 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:38,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:38,798 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:38,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:38,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:38,798 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:38,799 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:38,799 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:38,799 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:38,799 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:38,799 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:38,799 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:38,799 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:38,799 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:38,799 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:38,799 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:38,799 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:38,799 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:38,799 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:38,799 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,799 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:38,799 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:38,800 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,800 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:38,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:38,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:38,800 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:38,800 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:38,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:38,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:38,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:38,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:38,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:38,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15d1547e[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:38,801 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:38,801 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:38,802 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:38,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:38,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:38,802 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:38,802 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:38,802 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:38,802 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,802 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:38,803 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,803 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:38,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:38,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:38,803 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:38,803 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:38,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:38,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:38,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:38,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:38,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:38,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:38,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:38,804 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:38,805 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:38,805 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:38,805 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:38,805 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:38,805 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:38,805 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:38,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:38,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:38,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:38,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:38,806 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:38,806 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:38,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:38,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:38,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:38,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:38,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:38,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:38,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:38,807 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:38,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:38,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,808 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,808 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:38,808 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:38,808 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:38,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:38,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:38,809 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:38,809 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:38,809 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,809 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:38,809 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:38,809 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:38,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:38,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:38,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:38,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:38,809 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:38,809 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:38,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:38,840 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:38,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40616 deadline: 1681995338838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,840 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:38,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40568 deadline: 1681995338839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,840 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:38,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40570 deadline: 1681995338840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,909 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:38,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:38,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:38,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:38,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:38,913 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:38,913 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:38,913 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:38,913 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:38,913 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:38,913 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:38,914 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:38,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:38,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:38,914 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,914 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/edafa87cf153417081ded0dc9e2d9133 2023-04-20 12:54:38,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:38,914 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:38,915 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,915 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:38,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:38,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:38,915 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:38,915 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:38,915 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:38,915 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:38,915 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:38,915 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:38,919 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:38,919 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:38,919 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:38,919 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:38,919 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:38,919 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:38,919 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:38,919 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:38,920 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,920 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:38,920 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:38,921 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:38,921 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:38,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:38,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:38,922 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:38,922 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:38,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:38,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:38,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:38,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:38,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@559458cc[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:38,923 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:38,923 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:38,925 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,925 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:38,926 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:38,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:38,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:38,927 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:38,927 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:38,927 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:38,928 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:38,928 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,928 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:38,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:38,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:38,928 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:38,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:38,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:38,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:38,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:38,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:38,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:38,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:38,930 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:38,930 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:38,930 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:38,930 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:38,931 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:38,931 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:38,932 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:38,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:38,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:38,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:38,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:38,932 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:38,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:38,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:38,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:38,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:38,933 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:38,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:38,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:38,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:38,934 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:38,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:38,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,936 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:38,936 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:38,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:38,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:38,936 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,936 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:38,936 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:38,936 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:38,936 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:38,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:38,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:38,936 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:38,936 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:38,936 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:38,936 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:38,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:38,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:38,937 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:38,937 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:39,037 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:39,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:39,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:39,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:39,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:39,040 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:39,040 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:39,040 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:39,040 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:39,040 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,040 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:39,041 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:39,041 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:39,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:39,041 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,041 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:39,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:39,041 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:39,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:39,042 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:39,042 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:39,043 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:39,043 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:39,043 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:39,043 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:39,043 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:39,043 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:39,043 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:39,043 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:39,043 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:39,043 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:39,043 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:39,043 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:39,043 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,043 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:39,043 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:39,044 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,044 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:39,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:39,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:39,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:39,045 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:39,045 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:39,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:39,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:39,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:39,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:39,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71c36949[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:39,046 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:39,046 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:39,047 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:39,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:39,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:39,047 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:39,047 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:39,047 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:39,047 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,047 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:39,048 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:39,048 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:39,048 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:39,049 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:39,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:39,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:39,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:39,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:39,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:39,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:39,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:39,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:39,052 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:39,052 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:39,052 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:39,052 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:39,052 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:39,057 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,057 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,057 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:39,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:39,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:39,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:39,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:39,057 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:39,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:39,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:39,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:39,059 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:39,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:39,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,064 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,064 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:39,064 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:39,064 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:39,064 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:39,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:39,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:39,064 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:39,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:39,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:39,065 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,065 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:39,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:39,065 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:39,065 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:39,065 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:39,065 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:39,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:39,142 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:39,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40568 deadline: 1681995339141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,144 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:39,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40616 deadline: 1681995339143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:39,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40570 deadline: 1681995339143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,165 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:39,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:39,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:39,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:39,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:39,168 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:39,168 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:39,168 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:39,168 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:39,168 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,168 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:39,169 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:39,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:39,169 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:39,169 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,170 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:39,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:39,170 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:39,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:39,170 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:39,170 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:39,170 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:39,171 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:39,171 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:39,171 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:39,171 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:39,171 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:39,171 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:39,171 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:39,171 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:39,171 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:39,171 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:39,171 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:39,171 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,171 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:39,171 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:39,173 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:39,173 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:39,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:39,173 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:39,173 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:39,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:39,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:39,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:39,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:39,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:39,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31cd5b8c[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:39,175 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:39,175 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:39,176 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:39,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:39,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:39,176 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:39,176 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:39,176 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:39,176 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,176 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:39,177 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:39,177 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:39,177 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:39,177 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:39,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:39,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:39,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:39,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:39,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:39,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:39,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:39,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:39,180 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:39,181 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:39,181 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:39,181 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:39,181 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:39,182 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,182 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:39,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,182 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:39,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:39,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:39,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:39,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:39,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,183 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:39,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:39,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:39,184 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:39,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:39,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,185 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,185 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:39,185 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:39,185 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,185 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:39,185 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:39,185 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:39,185 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,185 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:39,186 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:39,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:39,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:39,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:39,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:39,189 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:39,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:39,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:39,189 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:39,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:39,197 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:39,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40560 deadline: 1681995339196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,203 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:39,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40554 deadline: 1681995339201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,290 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:39,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:39,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:39,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:39,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:39,292 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:39,292 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:39,292 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:39,292 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:39,293 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,293 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:39,293 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:39,293 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:39,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:39,294 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,294 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:39,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:39,294 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:39,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:39,294 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:39,294 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:39,295 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:39,295 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:39,295 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:39,295 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:39,295 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:39,295 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:39,295 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:39,295 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:39,295 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:39,295 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:39,295 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:39,295 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:39,295 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,295 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:39,295 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:39,296 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,296 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:39,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:39,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:39,297 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:39,297 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:39,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:39,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:39,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:39,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:39,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:39,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ea08b6f[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:39,298 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:39,298 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:39,299 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:39,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:39,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:39,299 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:39,299 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:39,299 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:39,299 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,299 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:39,299 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,299 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:39,300 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:39,300 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:39,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:39,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:39,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:39,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:39,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:39,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:39,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:39,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:39,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:39,302 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:39,302 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:39,302 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:39,302 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:39,302 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:39,303 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,303 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:39,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:39,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,303 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:39,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:39,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:39,304 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:39,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:39,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:39,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:39,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:39,305 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:39,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,307 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:39,307 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:39,307 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:39,307 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:39,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:39,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:39,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:39,307 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:39,307 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:39,307 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,307 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:39,307 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:39,307 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:39,307 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:39,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:39,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:39,307 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,326 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/0da123b9a3ff48bc8ba39fd7aedd686f 2023-04-20 12:54:39,407 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:39,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:39,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:39,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:39,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:39,410 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:39,410 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:39,410 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:39,410 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:39,411 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,411 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:39,412 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:39,412 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:39,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:39,412 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,412 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:39,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:39,412 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:39,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:39,413 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:39,413 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:39,413 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:39,413 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:39,413 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:39,413 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:39,413 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:39,413 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:39,413 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:39,413 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:39,413 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:39,414 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:39,414 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:39,414 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:39,414 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,414 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:39,414 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:39,414 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,414 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:39,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:39,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:39,415 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:39,415 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:39,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:39,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:39,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:39,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:39,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:39,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5905e44f[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:39,416 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:39,416 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:39,416 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:39,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:39,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:39,416 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:39,416 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:39,416 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:39,416 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,416 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:39,417 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:39,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:39,417 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:39,417 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:39,417 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:39,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:39,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:39,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:39,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:39,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:39,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:39,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:39,419 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:39,419 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:39,419 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:39,419 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:39,419 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:39,420 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,420 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,420 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:39,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:39,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:39,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,420 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:39,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:39,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:39,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:39,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:39,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:39,421 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:39,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:39,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,423 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,423 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:39,423 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:39,423 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:39,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:39,423 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:39,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:39,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:39,423 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:39,423 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:39,423 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:39,423 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:39,423 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:39,423 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:39,423 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:39,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:39,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:39,423 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,523 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:39,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:39,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:39,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:39,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:39,526 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:39,526 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:39,526 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:39,526 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:39,527 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,527 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:39,527 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:39,527 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:39,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:39,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:39,528 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,528 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:39,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:39,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:39,528 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:39,529 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:39,529 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:39,529 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:39,529 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:39,529 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:39,530 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:39,530 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:39,530 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:39,530 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:39,530 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:39,530 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:39,530 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:39,530 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:39,530 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,530 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:39,530 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:39,531 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,531 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:39,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:39,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:39,531 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:39,531 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:39,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:39,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:39,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:39,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:39,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:39,532 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:39,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33665707[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:39,533 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:39,533 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:39,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:39,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:39,533 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:39,533 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:39,533 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:39,533 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,533 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:39,534 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,534 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:39,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:39,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:39,534 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:39,534 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:39,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:39,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:39,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:39,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:39,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:39,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:39,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:39,536 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:39,536 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:39,536 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:39,536 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:39,536 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:39,537 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:39,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:39,537 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,538 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:39,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:39,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:39,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:39,538 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:39,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:39,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:39,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,539 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:39,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:39,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,540 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:39,541 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:39,541 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:39,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:39,541 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:39,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:39,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:39,541 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:39,541 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:39,541 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:39,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:39,541 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:39,541 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:39,541 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:39,541 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:39,541 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,641 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:39,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:39,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:39,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:39,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:39,644 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:39,644 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:39,644 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:39,644 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:39,645 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,645 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:39,646 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:39,646 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:39,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:39,646 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,646 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:39,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:39,646 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:39,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:39,646 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:39,647 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:39,647 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:39,647 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:39,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40568 deadline: 1681995339645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,647 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:39,647 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:39,647 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:39,647 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:39,647 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:39,647 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:39,647 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:39,647 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:39,647 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:39,647 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:39,647 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:39,647 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,647 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:39,647 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:39,648 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:39,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:39,648 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:39,649 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:39,649 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:39,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:39,649 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:39,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:39,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40616 deadline: 1681995339648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,649 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:39,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:39,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40570 deadline: 1681995339649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:39,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:39,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@682c7bdc[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:39,650 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:39,650 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:39,651 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:39,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:39,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:39,651 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:39,651 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:39,651 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:39,651 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,651 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:39,652 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:39,652 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,652 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:39,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:39,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:39,652 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:39,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:39,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:39,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:39,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:39,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:39,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:39,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:39,654 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:39,654 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:39,654 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:39,654 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:39,654 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:39,654 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:39,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:39,655 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,655 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:39,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:39,655 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:39,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:39,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:39,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:39,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:39,657 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:39,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:39,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,658 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:39,658 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:39,658 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:39,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:39,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:39,658 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:39,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:39,658 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:39,658 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:39,658 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:39,659 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:39,658 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:39,659 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:39,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:39,659 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:39,659 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:39,659 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,738 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/843d81ff59044f368098fc1f4b075949 2023-04-20 12:54:39,742 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/edafa87cf153417081ded0dc9e2d9133 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/edafa87cf153417081ded0dc9e2d9133 2023-04-20 12:54:39,746 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/edafa87cf153417081ded0dc9e2d9133, entries=150, sequenceid=162, filesize=11.8 K 2023-04-20 12:54:39,747 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/0da123b9a3ff48bc8ba39fd7aedd686f as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/0da123b9a3ff48bc8ba39fd7aedd686f 2023-04-20 12:54:39,756 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/0da123b9a3ff48bc8ba39fd7aedd686f, entries=150, sequenceid=162, filesize=11.8 K 2023-04-20 12:54:39,757 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/843d81ff59044f368098fc1f4b075949 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/843d81ff59044f368098fc1f4b075949 2023-04-20 12:54:39,759 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:39,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:39,761 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/843d81ff59044f368098fc1f4b075949, entries=150, sequenceid=162, filesize=11.8 K 2023-04-20 12:54:39,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:39,761 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~200.95 KB/205776, currentSize=107.34 KB/109920 for 969c768384097479d79029eb29913551 in 1254ms, sequenceid=162, compaction requested=true 2023-04-20 12:54:39,761 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:39,761 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:39,761 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:54:39,762 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:54:39,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:39,762 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:39,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:39,762 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:54:39,762 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:39,762 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:39,762 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:39,762 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:39,762 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,762 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:39,762 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35958 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:54:39,762 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35958 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:54:39,762 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): 969c768384097479d79029eb29913551/A is initiating minor compaction (all files) 2023-04-20 12:54:39,762 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): 969c768384097479d79029eb29913551/B is initiating minor compaction (all files) 2023-04-20 12:54:39,763 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 969c768384097479d79029eb29913551/A in TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:39,763 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 969c768384097479d79029eb29913551/B in TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:39,763 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/41beeff62f1c4175be854898a83d1cbd, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/7d4e295258d34a15a1d9ff50d2b65ad4, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/0da123b9a3ff48bc8ba39fd7aedd686f] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp, totalSize=35.1 K 2023-04-20 12:54:39,763 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/2ec5f1d068c749fab513bba61c5aaa9b, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/ee09fad504084976853fc119bbbebf14, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/edafa87cf153417081ded0dc9e2d9133] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp, totalSize=35.1 K 2023-04-20 12:54:39,763 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 41beeff62f1c4175be854898a83d1cbd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1681995277952 2023-04-20 12:54:39,763 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 2ec5f1d068c749fab513bba61c5aaa9b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1681995277952 2023-04-20 12:54:39,763 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,763 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:39,763 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 7d4e295258d34a15a1d9ff50d2b65ad4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1681995278454 2023-04-20 12:54:39,763 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting ee09fad504084976853fc119bbbebf14, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1681995278454 2023-04-20 12:54:39,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:39,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:39,764 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:39,764 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:39,764 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 0da123b9a3ff48bc8ba39fd7aedd686f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1681995278480 2023-04-20 12:54:39,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:39,764 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting edafa87cf153417081ded0dc9e2d9133, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1681995278480 2023-04-20 12:54:39,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:39,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:39,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:39,765 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:39,765 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:39,765 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:39,765 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:39,765 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:39,765 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:39,765 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:39,765 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:39,765 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:39,765 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:39,766 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 969c768384097479d79029eb29913551 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-20 12:54:39,766 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=A 2023-04-20 12:54:39,766 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:39,766 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=B 2023-04-20 12:54:39,766 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:39,766 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=C 2023-04-20 12:54:39,766 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:39,770 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): 969c768384097479d79029eb29913551#B#compaction#192 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:39,772 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 969c768384097479d79029eb29913551#A#compaction#191 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:39,787 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/09f3cd45a0ca4c6badd8ed43ab362fef as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/09f3cd45a0ca4c6badd8ed43ab362fef 2023-04-20 12:54:39,791 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 969c768384097479d79029eb29913551/A of 969c768384097479d79029eb29913551 into 09f3cd45a0ca4c6badd8ed43ab362fef(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:54:39,791 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:39,791 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551., storeName=969c768384097479d79029eb29913551/A, priority=13, startTime=1681995279761; duration=0sec 2023-04-20 12:54:39,791 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:54:39,791 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:54:39,792 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35958 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:54:39,792 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): 969c768384097479d79029eb29913551/C is initiating minor compaction (all files) 2023-04-20 12:54:39,793 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 969c768384097479d79029eb29913551/C in TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:39,793 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/56045c8f78ab4a9aa1fc117ae886e1e8, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/9d3c20565923423fbc69c1099c659957, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/843d81ff59044f368098fc1f4b075949] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp, totalSize=35.1 K 2023-04-20 12:54:39,793 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 56045c8f78ab4a9aa1fc117ae886e1e8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1681995277952 2023-04-20 12:54:39,793 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 9d3c20565923423fbc69c1099c659957, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1681995278454 2023-04-20 12:54:39,794 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 843d81ff59044f368098fc1f4b075949, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1681995278480 2023-04-20 12:54:39,805 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 969c768384097479d79029eb29913551#C#compaction#193 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:40,184 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/813b983bf05c425086c808bf1972f72d as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/813b983bf05c425086c808bf1972f72d 2023-04-20 12:54:40,187 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/ec66b7c5cfe14c33aee80948edb46eeb 2023-04-20 12:54:40,188 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 969c768384097479d79029eb29913551/B of 969c768384097479d79029eb29913551 into 813b983bf05c425086c808bf1972f72d(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:54:40,188 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:40,188 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551., storeName=969c768384097479d79029eb29913551/B, priority=13, startTime=1681995279761; duration=0sec 2023-04-20 12:54:40,188 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:40,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 969c768384097479d79029eb29913551 2023-04-20 12:54:40,213 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:40,223 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/f83daecfe8664c7b867a8091f1b32de6 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/f83daecfe8664c7b867a8091f1b32de6 2023-04-20 12:54:40,226 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/42d0d7bec070428a8b59b5914c0dab9c 2023-04-20 12:54:40,231 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 969c768384097479d79029eb29913551/C of 969c768384097479d79029eb29913551 into f83daecfe8664c7b867a8091f1b32de6(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:54:40,231 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:40,231 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551., storeName=969c768384097479d79029eb29913551/C, priority=13, startTime=1681995279762; duration=0sec 2023-04-20 12:54:40,231 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:40,232 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:40,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40560 deadline: 1681995340229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,233 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:40,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40554 deadline: 1681995340230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,252 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/cc0d805fe21747de9414a8fdfe621d31 2023-04-20 12:54:40,259 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/ec66b7c5cfe14c33aee80948edb46eeb as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/ec66b7c5cfe14c33aee80948edb46eeb 2023-04-20 12:54:40,264 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/ec66b7c5cfe14c33aee80948edb46eeb, entries=150, sequenceid=181, filesize=11.8 K 2023-04-20 12:54:40,266 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/42d0d7bec070428a8b59b5914c0dab9c as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/42d0d7bec070428a8b59b5914c0dab9c 2023-04-20 12:54:40,270 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/42d0d7bec070428a8b59b5914c0dab9c, entries=150, sequenceid=181, filesize=11.8 K 2023-04-20 12:54:40,271 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/cc0d805fe21747de9414a8fdfe621d31 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/cc0d805fe21747de9414a8fdfe621d31 2023-04-20 12:54:40,275 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/cc0d805fe21747de9414a8fdfe621d31, entries=150, sequenceid=181, filesize=11.8 K 2023-04-20 12:54:40,276 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 969c768384097479d79029eb29913551 in 510ms, sequenceid=181, compaction requested=false 2023-04-20 12:54:40,276 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:40,276 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:40,276 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:40,276 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:40,276 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,276 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:40,276 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:40,277 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:40,277 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:40,277 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,277 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:40,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:40,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:40,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:40,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:40,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:40,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:40,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:40,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42fd7e45[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:40,280 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:40,280 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:40,280 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:40,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:40,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:40,280 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:40,280 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:40,280 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:40,281 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,281 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:40,281 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:40,281 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,281 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:40,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:40,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:40,281 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:40,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:40,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:40,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:40,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:40,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:40,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:40,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:40,284 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:40,284 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:40,284 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:40,284 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:40,284 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:40,285 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,285 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:40,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:40,285 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:40,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:40,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:40,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:40,285 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:40,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:40,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:40,286 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:40,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:40,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,289 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,289 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:40,289 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:40,289 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,289 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:40,289 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:40,289 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:40,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:40,289 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:40,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:40,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:40,289 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:40,289 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:40,289 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,289 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:40,290 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:40,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:40,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:40,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:40,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 969c768384097479d79029eb29913551 2023-04-20 12:54:40,334 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 969c768384097479d79029eb29913551 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-20 12:54:40,335 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=A 2023-04-20 12:54:40,335 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:40,335 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=B 2023-04-20 12:54:40,335 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:40,335 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=C 2023-04-20 12:54:40,335 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:40,390 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:40,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:40,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:40,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:40,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:40,393 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:40,393 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:40,393 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:40,393 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:40,393 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,393 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:40,394 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:40,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40554 deadline: 1681995340391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,394 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:40,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40560 deadline: 1681995340391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,394 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:40,394 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:40,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:40,394 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,394 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:40,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:40,395 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:40,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:40,395 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:40,395 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:40,395 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:40,395 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:40,395 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:40,395 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:40,396 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:40,396 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:40,396 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:40,396 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:40,396 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:40,396 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:40,396 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:40,396 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:40,396 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,396 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:40,396 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:40,397 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:40,397 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:40,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:40,397 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:40,397 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:40,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:40,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:40,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:40,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:40,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:40,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@342ccdc9[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:40,398 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:40,398 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:40,399 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:40,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:40,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:40,399 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:40,399 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:40,399 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:40,399 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,399 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:40,400 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,400 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:40,400 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:40,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:40,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:40,400 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:40,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:40,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:40,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:40,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:40,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:40,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:40,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:40,402 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:40,402 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:40,402 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:40,402 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:40,402 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:40,403 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,403 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:40,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:40,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,403 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:40,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:40,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:40,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:40,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,404 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:40,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:40,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:40,404 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:40,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:40,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,406 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:40,406 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,406 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:40,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:40,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:40,406 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:40,406 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:40,406 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:40,406 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:40,406 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:40,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:40,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:40,407 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,407 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:40,407 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:40,407 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,407 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:40,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:40,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:40,496 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:40,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40554 deadline: 1681995340495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,496 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:40,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40560 deadline: 1681995340495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,507 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:40,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:40,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:40,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:40,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:40,510 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:40,510 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:40,510 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:40,510 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:40,510 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,510 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:40,511 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,511 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:40,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:40,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:40,511 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,511 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:40,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:40,512 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:40,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:40,512 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:40,512 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:40,512 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:40,512 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:40,512 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:40,512 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:40,512 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:40,513 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:40,513 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:40,513 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:40,513 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:40,513 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:40,513 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:40,513 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:40,513 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,513 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:40,513 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:40,514 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,514 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:40,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:40,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:40,514 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:40,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:40,514 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:40,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:40,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:40,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:40,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:40,515 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:40,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35f36a24[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:40,515 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:40,516 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,516 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:40,516 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:40,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:40,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:40,516 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:40,516 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:40,516 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:40,517 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:40,517 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:40,517 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:40,517 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:40,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:40,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:40,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:40,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:40,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:40,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:40,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:40,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:40,520 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:40,520 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:40,520 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:40,520 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:40,520 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:40,521 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,521 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:40,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:40,521 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:40,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:40,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:40,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:40,521 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:40,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:40,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:40,523 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:40,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:40,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,525 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:40,525 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:40,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:40,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:40,525 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:40,525 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:40,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:40,525 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:40,525 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:40,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:40,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:40,525 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:40,525 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,525 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:40,525 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:40,525 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,525 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:40,526 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:40,626 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:40,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:40,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:40,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:40,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:40,629 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:40,629 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:40,629 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:40,629 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:40,629 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,629 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:40,630 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:40,630 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:40,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:40,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:40,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,631 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,631 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:40,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:40,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:40,631 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:40,631 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:40,631 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:40,631 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:40,631 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:40,631 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:40,631 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:40,631 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:40,631 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:40,631 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:40,632 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:40,632 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:40,632 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:40,632 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:40,632 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,632 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:40,632 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:40,633 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,633 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:40,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:40,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:40,633 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:40,633 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:40,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:40,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:40,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:40,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:40,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:40,634 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:40,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e0a15dd[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:40,634 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:40,635 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:40,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:40,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:40,635 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:40,635 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:40,635 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:40,635 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,635 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:40,635 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,635 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:40,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:40,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:40,635 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:40,636 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:40,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:40,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:40,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:40,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:40,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:40,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:40,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:40,638 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:40,638 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:40,638 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:40,638 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:40,638 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:40,639 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:40,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:40,639 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,639 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:40,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:40,640 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:40,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:40,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:40,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:40,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:40,641 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:40,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:40,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,642 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,642 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:40,642 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:40,643 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,643 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:40,643 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:40,643 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:40,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:40,644 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:40,643 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:40,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:40,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:40,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:40,643 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:40,644 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,644 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:40,644 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:40,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:40,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:40,653 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:40,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40570 deadline: 1681995340652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,654 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:40,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40616 deadline: 1681995340653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,658 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:40,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40568 deadline: 1681995340657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,699 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:40,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40560 deadline: 1681995340697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,699 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:40,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40554 deadline: 1681995340698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,743 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/aa1a05911dc44fe79800e60ba76790c7 2023-04-20 12:54:40,744 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:40,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:40,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:40,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:40,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:40,749 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:40,749 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:40,749 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:40,749 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:40,749 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,749 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:40,751 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:40,751 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:40,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:40,751 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,751 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:40,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:40,752 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:40,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:40,752 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:40,752 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:40,752 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:40,752 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:40,752 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:40,752 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:40,753 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:40,753 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:40,753 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:40,753 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:40,753 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:40,753 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:40,753 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:40,753 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:40,753 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,753 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:40,753 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:40,754 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:40,754 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:40,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:40,754 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/41caef2949f140c18e523a83e5ba863e 2023-04-20 12:54:40,754 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:40,754 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:40,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:40,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:40,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:40,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:40,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:40,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17ac5b0f[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:40,756 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:40,756 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:40,757 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:40,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:40,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:40,757 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:40,757 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:40,757 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:40,758 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,758 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:40,758 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:40,758 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,758 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:40,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:40,758 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:40,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:40,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:40,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:40,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:40,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:40,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:40,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:40,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:40,762 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:40,762 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:40,762 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:40,762 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:40,762 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:40,763 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,763 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:40,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:40,763 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:40,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,764 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:40,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:40,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:40,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:40,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:40,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:40,765 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:40,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:40,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,766 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,766 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:40,766 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:40,766 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:40,766 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:40,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:40,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:40,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:40,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:40,766 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:40,767 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,767 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:40,767 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:40,767 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:40,767 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:40,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:40,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:40,767 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:40,767 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,867 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:40,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:40,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:40,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:40,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:40,870 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:40,870 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:40,870 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:40,870 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:40,870 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,870 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:40,871 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:40,871 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:40,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:40,871 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,871 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:40,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:40,871 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:40,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:40,872 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:40,872 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:40,872 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:40,872 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:40,872 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:40,872 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:40,872 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:40,872 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:40,872 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:40,872 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:40,872 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:40,872 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:40,872 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:40,872 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:40,872 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,873 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:40,873 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:40,873 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:40,873 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:40,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:40,874 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:40,874 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:40,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:40,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:40,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:40,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:40,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:40,875 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:40,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@453c9eb6[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:40,875 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:40,875 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:40,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:40,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:40,876 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:40,876 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:40,876 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:40,876 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,876 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:40,876 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,876 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:40,876 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:40,876 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:40,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:40,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:40,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:40,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:40,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:40,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:40,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:40,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:40,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:40,878 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:40,878 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:40,878 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:40,878 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:40,878 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:40,879 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,879 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,879 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:40,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:40,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:40,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:40,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:40,880 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:40,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:40,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:40,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:40,881 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:40,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:40,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,882 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:40,882 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:40,882 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,882 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:40,882 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:40,882 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,882 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:40,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:40,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:40,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:40,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:40,882 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:40,883 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:40,883 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,883 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:40,883 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:40,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:40,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:40,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:40,983 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:40,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:40,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:40,987 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-20 12:54:40,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:40,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:40,991 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:40,992 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:40,992 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:40,992 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:40,992 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,992 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:40,993 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:40,993 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:40,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:40,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:40,994 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:40,994 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:40,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:40,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:40,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:40,994 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:40,997 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:40,997 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:40,997 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:40,997 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:40,997 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:41,000 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:41,000 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:41,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40554 deadline: 1681995341000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40560 deadline: 1681995341000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,004 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:41,004 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:41,010 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:41,010 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:41,010 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:41,010 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:41,010 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:41,010 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:41,010 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,010 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:41,011 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:41,012 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,012 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:41,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:41,012 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,012 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:41,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:41,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:41,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:41,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:41,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:41,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bb06e7f[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:41,014 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:41,014 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,015 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,015 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:41,015 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,015 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:41,015 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:41,016 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:41,016 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,016 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:41,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:41,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:41,016 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:41,017 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:41,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:41,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:41,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:41,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:41,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:41,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:41,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:41,020 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:41,020 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:41,020 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:41,020 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:41,020 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:41,023 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,023 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:41,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:41,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,023 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:41,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:41,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:41,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:41,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,025 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:41,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:41,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:41,026 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:41,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:41,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,031 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:41,032 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:41,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:41,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:41,032 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,032 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:41,032 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:41,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:41,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:41,032 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:41,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:41,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:41,032 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,032 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,032 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:41,032 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,032 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:41,032 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,132 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:41,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:41,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:41,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:41,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:41,135 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:41,135 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:41,135 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:41,135 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:41,135 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,135 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:41,136 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,136 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:41,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:41,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:41,136 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,136 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:41,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:41,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:41,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:41,137 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:41,137 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:41,137 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:41,137 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:41,137 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:41,137 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:41,137 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:41,137 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:41,137 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:41,137 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:41,137 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:41,138 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:41,138 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:41,138 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:41,138 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,138 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:41,138 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:41,139 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,139 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:41,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:41,139 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,139 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:41,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:41,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:41,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:41,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:41,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:41,140 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:41,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fa2a29[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:41,140 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,141 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,141 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,141 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:41,141 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:41,141 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:41,142 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:41,142 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:41,142 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,142 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:41,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:41,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:41,142 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:41,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:41,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:41,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:41,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:41,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:41,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:41,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:41,144 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:41,144 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:41,144 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:41,144 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:41,144 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:41,145 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,145 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:41,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:41,145 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:41,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,146 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:41,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:41,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:41,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:41,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:41,147 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:41,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:41,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:41,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,149 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,149 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:41,149 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:41,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:41,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:41,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:41,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:41,149 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:41,149 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:41,149 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:41,149 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:41,149 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:41,149 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:41,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:41,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:41,149 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,150 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,150 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,150 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,164 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/a552cd724c7344ff8a423c633b647784 2023-04-20 12:54:41,168 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/aa1a05911dc44fe79800e60ba76790c7 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/aa1a05911dc44fe79800e60ba76790c7 2023-04-20 12:54:41,172 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/aa1a05911dc44fe79800e60ba76790c7, entries=150, sequenceid=198, filesize=11.8 K 2023-04-20 12:54:41,173 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/41caef2949f140c18e523a83e5ba863e as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/41caef2949f140c18e523a83e5ba863e 2023-04-20 12:54:41,176 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/41caef2949f140c18e523a83e5ba863e, entries=150, sequenceid=198, filesize=11.8 K 2023-04-20 12:54:41,177 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/a552cd724c7344ff8a423c633b647784 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/a552cd724c7344ff8a423c633b647784 2023-04-20 12:54:41,180 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/a552cd724c7344ff8a423c633b647784, entries=150, sequenceid=198, filesize=11.8 K 2023-04-20 12:54:41,181 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 969c768384097479d79029eb29913551 in 847ms, sequenceid=198, compaction requested=true 2023-04-20 12:54:41,181 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:41,181 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:41,181 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:54:41,181 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:54:41,181 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:54:41,181 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:54:41,182 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36211 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:54:41,182 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36211 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:54:41,182 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): 969c768384097479d79029eb29913551/A is initiating minor compaction (all files) 2023-04-20 12:54:41,182 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): 969c768384097479d79029eb29913551/B is initiating minor compaction (all files) 2023-04-20 12:54:41,182 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 969c768384097479d79029eb29913551/A in TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:41,182 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 969c768384097479d79029eb29913551/B in TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:41,182 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/09f3cd45a0ca4c6badd8ed43ab362fef, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/ec66b7c5cfe14c33aee80948edb46eeb, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/aa1a05911dc44fe79800e60ba76790c7] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp, totalSize=35.4 K 2023-04-20 12:54:41,182 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/813b983bf05c425086c808bf1972f72d, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/42d0d7bec070428a8b59b5914c0dab9c, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/41caef2949f140c18e523a83e5ba863e] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp, totalSize=35.4 K 2023-04-20 12:54:41,182 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 09f3cd45a0ca4c6badd8ed43ab362fef, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1681995278480 2023-04-20 12:54:41,182 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 813b983bf05c425086c808bf1972f72d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1681995278480 2023-04-20 12:54:41,183 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting ec66b7c5cfe14c33aee80948edb46eeb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1681995278528 2023-04-20 12:54:41,183 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 42d0d7bec070428a8b59b5914c0dab9c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1681995278528 2023-04-20 12:54:41,183 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting aa1a05911dc44fe79800e60ba76790c7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1681995280227 2023-04-20 12:54:41,183 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 41caef2949f140c18e523a83e5ba863e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1681995280227 2023-04-20 12:54:41,188 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): 969c768384097479d79029eb29913551#B#compaction#200 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:41,188 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 969c768384097479d79029eb29913551#A#compaction#199 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:41,200 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/a826d7ab90394257ade3dd452c0e8f9a as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/a826d7ab90394257ade3dd452c0e8f9a 2023-04-20 12:54:41,203 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 969c768384097479d79029eb29913551/A of 969c768384097479d79029eb29913551 into a826d7ab90394257ade3dd452c0e8f9a(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:54:41,203 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:41,203 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551., storeName=969c768384097479d79029eb29913551/A, priority=13, startTime=1681995281181; duration=0sec 2023-04-20 12:54:41,203 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:54:41,204 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:54:41,204 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36211 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:54:41,204 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): 969c768384097479d79029eb29913551/C is initiating minor compaction (all files) 2023-04-20 12:54:41,204 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 969c768384097479d79029eb29913551/C in TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:41,204 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/f83daecfe8664c7b867a8091f1b32de6, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/cc0d805fe21747de9414a8fdfe621d31, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/a552cd724c7344ff8a423c633b647784] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp, totalSize=35.4 K 2023-04-20 12:54:41,205 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting f83daecfe8664c7b867a8091f1b32de6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1681995278480 2023-04-20 12:54:41,205 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting cc0d805fe21747de9414a8fdfe621d31, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1681995278528 2023-04-20 12:54:41,205 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting a552cd724c7344ff8a423c633b647784, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1681995280227 2023-04-20 12:54:41,211 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 969c768384097479d79029eb29913551#C#compaction#201 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:41,249 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:41,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:41,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:41,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:41,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:41,252 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:41,252 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:41,252 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:41,252 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:41,252 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,252 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:41,253 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:41,253 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:41,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:41,253 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,253 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:41,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:41,254 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:41,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:41,254 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:41,254 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:41,254 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:41,254 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:41,254 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:41,255 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:41,259 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:41,259 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:41,259 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:41,259 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:41,259 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 969c768384097479d79029eb29913551 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-20 12:54:41,259 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=A 2023-04-20 12:54:41,259 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:41,259 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=B 2023-04-20 12:54:41,259 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:41,259 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=C 2023-04-20 12:54:41,260 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:41,266 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=218 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/00512d487f20498ab21ea438e7665422 2023-04-20 12:54:41,274 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=218 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/701bfda29614462a95ba3bf61fd418b8 2023-04-20 12:54:41,283 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=218 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/a0fa60a555ac49169fdf71c91157dbc0 2023-04-20 12:54:41,287 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/00512d487f20498ab21ea438e7665422 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/00512d487f20498ab21ea438e7665422 2023-04-20 12:54:41,293 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/00512d487f20498ab21ea438e7665422, entries=150, sequenceid=218, filesize=11.8 K 2023-04-20 12:54:41,294 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/701bfda29614462a95ba3bf61fd418b8 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/701bfda29614462a95ba3bf61fd418b8 2023-04-20 12:54:41,300 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/701bfda29614462a95ba3bf61fd418b8, entries=150, sequenceid=218, filesize=11.8 K 2023-04-20 12:54:41,301 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/a0fa60a555ac49169fdf71c91157dbc0 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/a0fa60a555ac49169fdf71c91157dbc0 2023-04-20 12:54:41,304 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/a0fa60a555ac49169fdf71c91157dbc0, entries=150, sequenceid=218, filesize=11.8 K 2023-04-20 12:54:41,304 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for 969c768384097479d79029eb29913551 in 45ms, sequenceid=218, compaction requested=false 2023-04-20 12:54:41,304 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:41,305 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:41,305 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:41,305 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:41,305 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,305 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:41,305 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:41,306 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,306 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:41,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:41,306 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:41,306 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:41,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:41,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:41,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:41,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:41,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70fb7dba[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:41,307 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:41,307 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,308 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,308 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:41,308 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:41,308 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:41,308 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,308 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:41,309 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:41,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:41,309 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:41,309 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:41,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:41,309 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:41,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:41,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:41,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:41,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:41,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:41,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:41,311 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:41,311 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:41,311 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:41,311 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:41,311 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:41,312 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,312 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:41,312 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:41,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:41,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,312 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:41,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:41,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:41,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:41,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:41,313 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:41,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:41,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:41,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,315 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:41,315 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:41,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:41,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:41,315 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,315 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:41,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:41,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:41,315 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:41,315 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:41,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:41,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:41,315 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:41,315 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:41,316 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:41,315 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,316 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,316 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,316 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,416 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:41,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:41,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:41,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:41,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:41,418 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:41,418 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:41,418 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:41,418 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:41,419 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,419 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:41,419 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:41,419 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:41,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:41,420 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,420 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:41,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:41,420 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:41,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:41,420 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:41,420 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:41,420 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:41,420 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:41,421 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:41,421 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:41,421 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:41,421 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:41,421 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:41,421 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:41,421 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:41,421 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:41,421 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:41,421 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,421 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:41,421 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:41,422 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,422 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:41,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:41,422 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,422 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:41,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:41,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:41,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:41,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:41,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:41,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5113943d[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:41,423 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:41,423 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,424 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,424 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:41,424 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,424 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:41,424 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:41,424 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:41,425 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,425 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:41,425 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:41,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:41,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:41,425 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:41,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:41,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:41,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:41,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:41,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:41,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:41,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:41,427 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:41,427 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:41,427 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:41,427 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:41,427 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:41,428 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,428 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:41,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:41,428 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:41,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:41,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:41,428 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:41,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:41,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:41,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:41,429 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:41,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:41,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,431 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,431 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:41,431 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:41,431 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:41,431 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,431 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,431 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,431 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:41,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:41,431 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:41,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:41,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:41,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:41,431 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:41,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:41,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:41,432 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:41,432 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:41,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 969c768384097479d79029eb29913551 2023-04-20 12:54:41,510 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 969c768384097479d79029eb29913551 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-20 12:54:41,511 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=A 2023-04-20 12:54:41,511 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:41,511 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=B 2023-04-20 12:54:41,511 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:41,511 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=C 2023-04-20 12:54:41,511 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:41,532 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:41,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:41,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:41,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:41,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:41,534 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:41,534 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:41,534 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:41,534 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:41,535 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,535 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:41,535 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:41,535 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:41,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:41,536 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,536 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:41,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:41,536 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:41,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:41,536 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:41,536 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:41,536 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:41,536 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:41,536 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:41,536 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:41,537 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:41,537 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:41,537 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:41,537 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:41,537 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:41,537 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:41,537 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:41,537 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:41,537 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,537 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:41,537 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:41,538 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,538 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:41,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:41,538 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,538 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:41,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:41,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:41,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:41,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:41,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:41,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39b8cbf6[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:41,539 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:41,539 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,540 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,540 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:41,540 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,540 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:41,540 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:41,540 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:41,541 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:41,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:41,542 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:41,542 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:41,542 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:41,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:41,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:41,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:41,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:41,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:41,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:41,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:41,543 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:41,544 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:41,544 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:41,544 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:41,544 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:41,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681995281533, max timestamp=1681995281543Num uniques 859; 2023-04-20 12:54:41,545 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 12:54:41,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681995281533, max timestamp=1681995281543Num uniques 859; 2023-04-20 12:54:41,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 12:54:41,545 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,545 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681995281533, max timestamp=1681995281543Num uniques 859; 2023-04-20 12:54:41,546 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 12:54:41,545 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,545 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:41,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:41,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:41,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:41,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:41,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:41,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:41,547 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:41,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:41,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:41,549 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:41,549 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:41,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:41,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:41,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:41,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:41,550 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,550 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:41,550 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:41,550 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,550 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:41,550 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,550 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,550 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:41,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:41,551 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:41,551 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:41,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:41,576 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:41,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40560 deadline: 1681995341573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,579 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:41,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40554 deadline: 1681995341576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,600 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/9c97fc731af945ee90e24f2f21754a5b as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/9c97fc731af945ee90e24f2f21754a5b 2023-04-20 12:54:41,604 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 969c768384097479d79029eb29913551/B of 969c768384097479d79029eb29913551 into 9c97fc731af945ee90e24f2f21754a5b(size=12.0 K), total size for store is 23.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:54:41,604 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:41,604 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551., storeName=969c768384097479d79029eb29913551/B, priority=13, startTime=1681995281181; duration=0sec 2023-04-20 12:54:41,604 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:41,619 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/ce233aa0bbad4d9387ba032d6c7b099a as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/ce233aa0bbad4d9387ba032d6c7b099a 2023-04-20 12:54:41,624 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 969c768384097479d79029eb29913551/C of 969c768384097479d79029eb29913551 into ce233aa0bbad4d9387ba032d6c7b099a(size=12.0 K), total size for store is 23.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:54:41,624 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:41,625 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551., storeName=969c768384097479d79029eb29913551/C, priority=13, startTime=1681995281181; duration=0sec 2023-04-20 12:54:41,625 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:41,651 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:41,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:41,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:41,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:41,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:41,654 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:41,654 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:41,654 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:41,654 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:41,655 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,655 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:41,656 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,656 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:41,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:41,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:41,656 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,656 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:41,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:41,656 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:41,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:41,657 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:41,657 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:41,657 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:41,657 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:41,657 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:41,657 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:41,658 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:41,658 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:41,658 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:41,658 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:41,658 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:41,658 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:41,658 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:41,658 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:41,658 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,658 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:41,658 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:41,659 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,659 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:41,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:41,660 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,660 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:41,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:41,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:41,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:41,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:41,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:41,661 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:41,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6dc7058a[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:41,661 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,662 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,662 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,662 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:41,663 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:41,663 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:41,663 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:41,663 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:41,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:41,664 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:41,664 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:41,664 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:41,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:41,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:41,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:41,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:41,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:41,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:41,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:41,666 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:41,666 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:41,666 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:41,666 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:41,666 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:41,667 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:41,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:41,668 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,668 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:41,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:41,668 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:41,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:41,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:41,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:41,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:41,669 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:41,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:41,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,671 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:41,671 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:41,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:41,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:41,671 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,671 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:41,671 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:41,671 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,671 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:41,671 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:41,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:41,671 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,671 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:41,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:41,672 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:41,672 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:41,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:41,679 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:41,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40560 deadline: 1681995341677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,683 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:41,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40554 deadline: 1681995341680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,772 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:41,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:41,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:41,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:41,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:41,775 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:41,775 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:41,775 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:41,775 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:41,775 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,775 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:41,776 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:41,776 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:41,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:41,776 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,776 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:41,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:41,777 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:41,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:41,777 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:41,777 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:41,777 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:41,777 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:41,777 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:41,777 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:41,777 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:41,777 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:41,777 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:41,777 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:41,778 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:41,778 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:41,778 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:41,778 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:41,778 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,778 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:41,778 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:41,779 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,779 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:41,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:41,779 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,779 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:41,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:41,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:41,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:41,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:41,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:41,780 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:41,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@467aa325[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:41,780 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,781 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,782 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:41,782 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:41,782 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:41,782 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,782 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:41,782 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,782 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:41,782 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:41,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:41,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:41,783 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:41,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:41,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:41,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:41,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:41,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:41,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:41,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:41,785 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:41,785 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:41,785 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:41,785 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:41,785 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:41,786 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,786 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:41,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,786 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:41,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:41,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:41,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:41,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:41,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,788 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:41,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:41,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:41,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,788 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:41,790 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:41,790 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:41,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:41,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:41,790 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:41,790 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:41,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:41,790 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:41,790 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:41,790 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,790 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,790 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,790 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,790 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:41,790 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:41,790 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:41,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:41,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:41,881 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:41,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40560 deadline: 1681995341880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,886 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:41,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40554 deadline: 1681995341885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,891 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:41,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:41,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:41,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:41,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:41,893 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:41,893 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:41,893 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:41,893 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:41,893 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,893 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:41,894 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,894 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:41,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:41,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:41,894 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,894 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:41,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:41,895 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:41,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:41,895 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:41,895 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:41,895 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:41,895 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:41,895 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:41,895 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:41,896 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:41,896 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:41,896 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:41,896 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:41,896 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:41,896 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:41,896 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:41,896 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:41,896 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,896 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:41,896 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:41,897 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,897 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:41,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:41,897 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,897 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:41,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:41,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:41,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:41,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:41,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:41,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b0565ab[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:41,898 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:41,898 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,899 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,899 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:41,899 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,899 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:41,899 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:41,899 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:41,900 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,900 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:41,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:41,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:41,900 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:41,900 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:41,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:41,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:41,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:41,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:41,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:41,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:41,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:41,902 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:41,902 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:41,902 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:41,902 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:41,902 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:41,903 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,903 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,903 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:41,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:41,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:41,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:41,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:41,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:41,904 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:41,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:41,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:41,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,905 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:41,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:41,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,907 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,907 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:41,907 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:41,907 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:41,907 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:41,907 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:41,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:41,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:41,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:41,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:41,907 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:41,907 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:41,907 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,907 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:41,907 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:41,907 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:41,908 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:41,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:41,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:41,918 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=229 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/5818b8b12bfa4563a389a62871f9d4f1 2023-04-20 12:54:42,008 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:42,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:42,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:42,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:42,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:42,010 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:42,010 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:42,010 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:42,010 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:42,011 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,011 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:42,011 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:42,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:42,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:42,011 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:42,012 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,012 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:42,012 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:42,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:42,012 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:42,012 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:42,012 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:42,013 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:42,013 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:42,014 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:42,014 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:42,014 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:42,014 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:42,014 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:42,014 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:42,014 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:42,014 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:42,014 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:42,014 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,014 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:42,014 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:42,015 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,015 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:42,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:42,015 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,015 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:42,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:42,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:42,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:42,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:42,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:42,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@741de846[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:42,016 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:42,016 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,017 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,017 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:42,017 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,017 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:42,017 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:42,017 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:42,018 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:42,018 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,018 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:42,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:42,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:42,018 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:42,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:42,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:42,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:42,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:42,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:42,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:42,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:42,020 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:42,020 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:42,020 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:42,020 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:42,020 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:42,021 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,021 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:42,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:42,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,021 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:42,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,021 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:42,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:42,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:42,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:42,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:42,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:42,022 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:42,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:42,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,024 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,024 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:42,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:42,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:42,024 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:42,024 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:42,024 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:42,024 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:42,024 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,024 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:42,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:42,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:42,024 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:42,024 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:42,024 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,024 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,024 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:42,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:42,124 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:42,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:42,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:42,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:42,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:42,127 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:42,127 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:42,127 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:42,127 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:42,127 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,127 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:42,128 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:42,128 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:42,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:42,128 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,128 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:42,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:42,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:42,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:42,129 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:42,129 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:42,129 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:42,129 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:42,129 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:42,129 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:42,129 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:42,129 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:42,129 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:42,129 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:42,130 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:42,130 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:42,130 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:42,130 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:42,130 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,130 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:42,130 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:42,130 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,130 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:42,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:42,131 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,131 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:42,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:42,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:42,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:42,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:42,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:42,132 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:42,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2123581a[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:42,132 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,132 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,132 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:42,132 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:42,132 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:42,132 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,132 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:42,133 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,133 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:42,133 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:42,133 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:42,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:42,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:42,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:42,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:42,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:42,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:42,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:42,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:42,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:42,135 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:42,135 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:42,135 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:42,135 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:42,135 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:42,136 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,136 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:42,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:42,136 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:42,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:42,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:42,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:42,136 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:42,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:42,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:42,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,137 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:42,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:42,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,139 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:42,139 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:42,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:42,139 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:42,139 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:42,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:42,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:42,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:42,139 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:42,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:42,139 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:42,139 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,139 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:42,139 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,139 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,140 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,139 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:42,184 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:42,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40560 deadline: 1681995342183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,188 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:42,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40554 deadline: 1681995342187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,240 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:42,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:42,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:42,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:42,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:42,243 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:42,243 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:42,243 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:42,243 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:42,243 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,243 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:42,244 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,244 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:42,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:42,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:42,244 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,244 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:42,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:42,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:42,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:42,244 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:42,245 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:42,245 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:42,245 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:42,245 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:42,245 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:42,245 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:42,245 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:42,245 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:42,245 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:42,245 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:42,245 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:42,245 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:42,245 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:42,245 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,245 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:42,245 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:42,246 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,246 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:42,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:42,246 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,246 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:42,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:42,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:42,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:42,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:42,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:42,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32d3a490[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:42,247 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:42,247 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,248 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,248 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:42,248 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:42,248 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:42,248 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,248 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:42,249 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,249 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:42,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:42,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:42,249 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:42,249 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:42,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:42,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:42,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:42,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:42,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:42,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:42,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:42,251 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:42,251 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:42,251 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:42,251 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:42,251 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:42,252 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,252 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:42,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:42,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,252 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:42,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:42,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:42,252 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:42,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:42,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:42,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:42,253 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:42,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:42,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,255 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,255 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:42,255 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:42,255 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:42,255 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:42,255 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:42,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:42,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:42,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:42,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:42,255 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:42,255 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,255 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,255 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,255 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:42,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:42,255 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:42,255 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:42,330 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=229 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/6c31345bb55f4528a8557d551e36fda9 2023-04-20 12:54:42,355 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:42,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:42,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:42,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:42,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:42,358 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:42,358 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:42,358 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:42,358 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:42,358 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,358 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:42,359 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:42,359 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:42,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:42,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:42,360 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,360 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:42,360 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:42,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:42,360 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:42,360 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:42,360 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:42,360 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:42,360 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:42,360 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:42,361 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:42,361 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:42,361 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:42,361 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:42,361 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:42,361 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:42,361 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:42,361 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:42,361 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,361 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:42,361 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:42,362 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,362 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:42,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:42,362 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,362 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:42,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:42,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:42,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:42,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:42,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:42,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@125bb75c[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:42,364 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:42,364 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,364 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,365 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:42,365 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:42,365 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,365 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:42,365 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:42,365 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:42,365 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:42,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:42,366 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:42,366 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:42,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:42,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:42,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:42,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:42,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:42,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:42,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:42,367 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:42,367 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:42,367 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:42,367 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:42,367 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:42,369 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,369 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:42,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:42,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,369 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:42,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:42,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:42,369 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:42,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:42,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:42,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:42,370 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:42,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:42,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,372 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:42,372 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:42,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:42,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:42,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:42,372 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:42,372 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:42,372 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:42,372 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,372 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,372 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,372 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:42,372 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:42,372 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:42,373 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:42,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:42,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:42,473 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:42,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:42,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:42,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:42,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:42,476 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:42,476 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:42,476 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:42,476 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:42,476 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,476 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:42,477 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:42,477 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:42,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:42,477 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,477 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:42,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:42,477 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:42,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:42,478 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:42,478 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:42,478 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:42,478 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:42,478 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:42,478 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:42,478 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:42,478 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:42,478 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:42,478 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:42,478 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:42,478 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:42,478 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:42,478 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:42,478 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,478 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:42,478 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:42,479 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,479 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:42,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:42,480 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,480 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:42,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:42,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:42,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:42,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:42,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:42,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@99f2289[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:42,481 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:42,481 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,481 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,482 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:42,482 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:42,482 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:42,482 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,482 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:42,483 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,483 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:42,483 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:42,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:42,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:42,483 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:42,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:42,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:42,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:42,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:42,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:42,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:42,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:42,485 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:42,485 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:42,485 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:42,485 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:42,485 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:42,486 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,486 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,486 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:42,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:42,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:42,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:42,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:42,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:42,486 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:42,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:42,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:42,487 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:42,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:42,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,489 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,489 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:42,489 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:42,489 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,489 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:42,489 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:42,489 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:42,489 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:42,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:42,489 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,489 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:42,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:42,489 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:42,489 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:42,489 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:42,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:42,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:42,589 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:42,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:42,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:42,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:42,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:42,592 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:42,592 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:42,593 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:42,593 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:42,593 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,593 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:42,594 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,594 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:42,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:42,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:42,594 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,594 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:42,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:42,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:42,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:42,595 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:42,595 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:42,595 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:42,595 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:42,595 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:42,595 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:42,595 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:42,595 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:42,595 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:42,595 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:42,595 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:42,595 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:42,595 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:42,596 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:42,596 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,596 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:42,596 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:42,596 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,596 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:42,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:42,597 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,597 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:42,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:42,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:42,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:42,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:42,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:42,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32c74828[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:42,598 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:42,598 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,599 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,599 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:42,599 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:42,599 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,599 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:42,599 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:42,600 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,600 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:42,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:42,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:42,600 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:42,600 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:42,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:42,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:42,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:42,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:42,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:42,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:42,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:42,602 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:42,602 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:42,602 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:42,602 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:42,602 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:42,603 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,603 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:42,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,603 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:42,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:42,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:42,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:42,604 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:42,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:42,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:42,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:42,604 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:42,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:42,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,607 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,607 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:42,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:42,607 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:42,607 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:42,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:42,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:42,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:42,607 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:42,607 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:42,607 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,607 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,607 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,607 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:42,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:42,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:42,607 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,607 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:42,608 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:42,668 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:42,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40616 deadline: 1681995342665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,670 DEBUG [Thread-1494] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4139 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551., hostname=jenkins-hbase20.apache.org,36277,1681995213947, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-20 12:54:42,671 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:42,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40568 deadline: 1681995342669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,673 DEBUG [Thread-1492] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4141 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551., hostname=jenkins-hbase20.apache.org,36277,1681995213947, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-20 12:54:42,675 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:42,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40570 deadline: 1681995342673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,677 DEBUG [Thread-1488] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4144 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551., hostname=jenkins-hbase20.apache.org,36277,1681995213947, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-20 12:54:42,691 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:42,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40560 deadline: 1681995342690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,691 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:42,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40554 deadline: 1681995342690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,708 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:42,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:42,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:42,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:42,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:42,710 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:42,710 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:42,710 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:42,710 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:42,711 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,711 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:42,712 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,712 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:42,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:42,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:42,712 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,712 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:42,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:42,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:42,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:42,713 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:42,713 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:42,713 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:42,713 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:42,713 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:42,713 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:42,713 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:42,713 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:42,713 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:42,714 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:42,714 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:42,714 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:42,714 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:42,714 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:42,714 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,714 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:42,714 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:42,715 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,715 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:42,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:42,715 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:42,715 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:42,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:42,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:42,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:42,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:42,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d9f8dc4[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:42,717 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:42,717 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,717 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,717 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:42,717 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:42,717 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:42,717 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,717 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:42,718 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:42,718 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:42,718 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,718 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:42,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:42,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:42,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:42,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:42,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:42,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:42,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:42,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:42,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:42,721 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:42,721 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:42,721 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:42,721 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:42,721 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:42,722 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,722 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:42,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:42,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,722 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:42,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:42,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:42,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:42,723 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:42,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:42,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:42,723 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:42,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:42,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,725 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,725 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:42,725 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:42,725 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:42,725 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:42,725 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:42,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:42,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:42,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:42,725 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:42,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:42,726 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,726 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,726 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:42,726 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,726 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:42,726 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:42,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:42,747 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=229 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/73e8627ea7e249e9808aad7cd18906e6 2023-04-20 12:54:42,751 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/5818b8b12bfa4563a389a62871f9d4f1 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/5818b8b12bfa4563a389a62871f9d4f1 2023-04-20 12:54:42,754 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/5818b8b12bfa4563a389a62871f9d4f1, entries=200, sequenceid=229, filesize=14.1 K 2023-04-20 12:54:42,755 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/6c31345bb55f4528a8557d551e36fda9 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/6c31345bb55f4528a8557d551e36fda9 2023-04-20 12:54:42,759 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/6c31345bb55f4528a8557d551e36fda9, entries=150, sequenceid=229, filesize=11.8 K 2023-04-20 12:54:42,759 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/73e8627ea7e249e9808aad7cd18906e6 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/73e8627ea7e249e9808aad7cd18906e6 2023-04-20 12:54:42,762 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/73e8627ea7e249e9808aad7cd18906e6, entries=150, sequenceid=229, filesize=11.8 K 2023-04-20 12:54:42,763 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 969c768384097479d79029eb29913551 in 1253ms, sequenceid=229, compaction requested=true 2023-04-20 12:54:42,763 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:42,763 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:42,763 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:54:42,763 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:54:42,763 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:54:42,763 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:54:42,764 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38704 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:54:42,764 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:54:42,764 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): 969c768384097479d79029eb29913551/A is initiating minor compaction (all files) 2023-04-20 12:54:42,764 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): 969c768384097479d79029eb29913551/B is initiating minor compaction (all files) 2023-04-20 12:54:42,764 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 969c768384097479d79029eb29913551/A in TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:42,764 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 969c768384097479d79029eb29913551/B in TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:42,764 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/a826d7ab90394257ade3dd452c0e8f9a, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/00512d487f20498ab21ea438e7665422, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/5818b8b12bfa4563a389a62871f9d4f1] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp, totalSize=37.8 K 2023-04-20 12:54:42,764 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/9c97fc731af945ee90e24f2f21754a5b, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/701bfda29614462a95ba3bf61fd418b8, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/6c31345bb55f4528a8557d551e36fda9] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp, totalSize=35.5 K 2023-04-20 12:54:42,765 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting a826d7ab90394257ade3dd452c0e8f9a, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1681995280227 2023-04-20 12:54:42,765 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 9c97fc731af945ee90e24f2f21754a5b, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1681995280227 2023-04-20 12:54:42,765 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 00512d487f20498ab21ea438e7665422, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1681995280359 2023-04-20 12:54:42,765 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 701bfda29614462a95ba3bf61fd418b8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1681995280359 2023-04-20 12:54:42,765 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 6c31345bb55f4528a8557d551e36fda9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1681995281507 2023-04-20 12:54:42,766 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 5818b8b12bfa4563a389a62871f9d4f1, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1681995281504 2023-04-20 12:54:42,770 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): 969c768384097479d79029eb29913551#B#compaction#208 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:42,771 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 969c768384097479d79029eb29913551#A#compaction#209 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:42,786 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/0daed446938d4efd8813719b59573423 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/0daed446938d4efd8813719b59573423 2023-04-20 12:54:42,787 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/2735ca4481314a9e935dbd7c3529f5cb as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/2735ca4481314a9e935dbd7c3529f5cb 2023-04-20 12:54:42,791 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 969c768384097479d79029eb29913551/A of 969c768384097479d79029eb29913551 into 0daed446938d4efd8813719b59573423(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:54:42,791 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:42,791 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551., storeName=969c768384097479d79029eb29913551/A, priority=13, startTime=1681995282763; duration=0sec 2023-04-20 12:54:42,791 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:54:42,791 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:54:42,791 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 969c768384097479d79029eb29913551/B of 969c768384097479d79029eb29913551 into 2735ca4481314a9e935dbd7c3529f5cb(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:54:42,791 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:42,791 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551., storeName=969c768384097479d79029eb29913551/B, priority=13, startTime=1681995282763; duration=0sec 2023-04-20 12:54:42,791 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:42,792 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:54:42,792 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): 969c768384097479d79029eb29913551/C is initiating minor compaction (all files) 2023-04-20 12:54:42,792 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 969c768384097479d79029eb29913551/C in TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:42,792 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/ce233aa0bbad4d9387ba032d6c7b099a, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/a0fa60a555ac49169fdf71c91157dbc0, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/73e8627ea7e249e9808aad7cd18906e6] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp, totalSize=35.5 K 2023-04-20 12:54:42,792 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting ce233aa0bbad4d9387ba032d6c7b099a, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1681995280227 2023-04-20 12:54:42,793 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting a0fa60a555ac49169fdf71c91157dbc0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1681995280359 2023-04-20 12:54:42,793 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 73e8627ea7e249e9808aad7cd18906e6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1681995281507 2023-04-20 12:54:42,797 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 969c768384097479d79029eb29913551#C#compaction#210 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:42,804 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/f1379ca7ad0f4f28a8ca4b59192cd475 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/f1379ca7ad0f4f28a8ca4b59192cd475 2023-04-20 12:54:42,808 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 969c768384097479d79029eb29913551/C of 969c768384097479d79029eb29913551 into f1379ca7ad0f4f28a8ca4b59192cd475(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:54:42,808 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:42,808 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551., storeName=969c768384097479d79029eb29913551/C, priority=13, startTime=1681995282763; duration=0sec 2023-04-20 12:54:42,808 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:42,826 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:42,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:42,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:42,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:42,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:42,829 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:42,829 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:42,829 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:42,829 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:42,829 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,829 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:42,830 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,830 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:42,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:42,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:42,830 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,830 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:42,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:42,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:42,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:42,830 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:42,831 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:42,831 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:42,831 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:42,831 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:42,831 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:42,831 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:42,831 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:42,831 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:42,831 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:42,831 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 969c768384097479d79029eb29913551 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-20 12:54:42,831 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=A 2023-04-20 12:54:42,831 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:54:42,831 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=B 2023-04-20 12:54:42,832 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:54:42,832 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=C 2023-04-20 12:54:42,832 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:54:42,843 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/7f9679f37bd1408099826e5df79a3cda 2023-04-20 12:54:42,855 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/d654be6e03524d82802a4211c680416f 2023-04-20 12:54:42,864 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/00250837fd23488c908cec25c82873f4 2023-04-20 12:54:42,868 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/7f9679f37bd1408099826e5df79a3cda as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/7f9679f37bd1408099826e5df79a3cda 2023-04-20 12:54:42,872 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/7f9679f37bd1408099826e5df79a3cda, entries=150, sequenceid=271, filesize=11.9 K 2023-04-20 12:54:42,873 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/d654be6e03524d82802a4211c680416f as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/d654be6e03524d82802a4211c680416f 2023-04-20 12:54:42,882 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/d654be6e03524d82802a4211c680416f, entries=150, sequenceid=271, filesize=11.9 K 2023-04-20 12:54:42,883 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/00250837fd23488c908cec25c82873f4 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/00250837fd23488c908cec25c82873f4 2023-04-20 12:54:42,887 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/00250837fd23488c908cec25c82873f4, entries=150, sequenceid=271, filesize=11.9 K 2023-04-20 12:54:42,888 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=0 B/0 for 969c768384097479d79029eb29913551 in 57ms, sequenceid=271, compaction requested=false 2023-04-20 12:54:42,888 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:42,888 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:42,888 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:42,888 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:42,888 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,888 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:42,888 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:42,889 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,889 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:42,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:42,889 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:42,889 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:42,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:42,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:42,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:42,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:42,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5231387c[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:42,891 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:42,891 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,891 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,892 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:42,892 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:42,892 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:42,892 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,892 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:42,892 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,892 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:42,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:42,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:42,892 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:42,892 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:42,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:42,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:42,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:42,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:42,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:42,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:42,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:42,894 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:42,894 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:42,894 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:42,894 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:42,895 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:42,895 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,895 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,895 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:42,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:42,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:42,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:42,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:42,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:42,896 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:42,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:42,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:42,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:42,897 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:42,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:42,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,899 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:42,899 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:42,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:42,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:42,899 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:42,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:42,899 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:42,899 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:42,899 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:42,899 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:42,899 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:42,899 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,899 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:42,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:42,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:42,899 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:42,899 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:42,899 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,000 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:43,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:43,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:43,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:43,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:43,002 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:43,002 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:43,003 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:43,003 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:43,003 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,003 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:43,003 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,003 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:43,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:43,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:43,004 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,004 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:43,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:43,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:43,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:43,004 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:43,005 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:43,005 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:43,005 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:43,005 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:43,005 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:43,005 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:43,005 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:43,005 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:43,005 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:43,005 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:43,006 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:43,006 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:43,006 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:43,006 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,006 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:43,006 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:43,007 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,007 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:43,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:43,007 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,007 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:43,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:43,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:43,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:43,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:43,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@224ad3d9[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:43,008 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:43,008 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,009 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,009 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:43,009 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:43,009 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,009 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:43,009 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:43,010 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:43,010 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,010 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:43,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:43,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:43,010 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:43,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:43,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:43,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:43,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:43,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:43,012 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:43,012 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:43,012 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:43,012 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:43,012 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:43,013 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,013 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,013 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:43,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:43,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:43,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:43,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:43,013 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:43,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:43,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,014 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:43,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,016 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:43,016 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:43,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:43,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:43,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:43,016 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,016 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:43,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:43,016 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:43,016 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:43,016 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:43,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:43,016 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:43,016 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,016 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,016 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,016 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:43,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:43,116 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:43,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:43,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:43,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:43,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:43,120 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:43,120 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:43,120 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:43,120 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:43,120 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,120 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:43,121 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:43,121 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:43,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:43,121 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,121 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:43,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:43,121 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:43,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:43,122 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:43,122 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:43,122 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:43,122 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:43,122 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:43,122 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:43,122 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:43,122 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:43,122 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:43,122 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:43,123 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:43,123 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:43,123 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:43,123 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:43,123 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,123 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:43,123 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:43,124 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,124 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:43,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:43,124 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,125 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:43,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:43,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:43,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:43,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:43,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a4449b0[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:43,126 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:43,126 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,127 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,127 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:43,127 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:43,127 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:43,128 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,128 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:43,128 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,128 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:43,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:43,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:43,128 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:43,129 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:43,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:43,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:43,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:43,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:43,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:43,131 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:43,131 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:43,131 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:43,131 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:43,131 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:43,132 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:43,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:43,132 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,132 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:43,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:43,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:43,133 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:43,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:43,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,134 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:43,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,135 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:43,135 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:43,135 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:43,135 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:43,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:43,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:43,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:43,135 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:43,136 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:43,136 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,136 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,136 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,136 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:43,136 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:43,136 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:43,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:43,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:43,236 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:43,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:43,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:43,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:43,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:43,242 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:43,242 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:43,242 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:43,242 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:43,243 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,243 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:43,244 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,244 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:43,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:43,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:43,244 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,244 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:43,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:43,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:43,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:43,245 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:43,247 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:43,247 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:43,247 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:43,247 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:43,247 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:43,248 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:43,249 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:43,249 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:43,249 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:43,249 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:43,249 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:43,249 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:43,249 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:43,249 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,249 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:43,249 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:43,256 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,256 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:43,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:43,256 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,256 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:43,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:43,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:43,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:43,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:43,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b541091[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:43,257 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:43,257 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,258 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,258 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:43,258 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:43,258 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:43,258 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,258 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:43,259 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,259 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:43,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:43,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:43,259 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:43,259 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:43,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:43,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:43,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:43,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:43,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:43,261 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:43,261 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:43,261 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:43,261 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:43,261 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:43,263 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,263 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:43,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:43,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,263 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:43,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,263 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:43,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:43,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:43,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:43,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,264 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:43,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,266 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,266 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:43,266 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:43,266 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:43,266 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:43,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:43,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:43,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:43,266 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:43,266 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:43,266 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,266 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:43,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:43,266 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:43,266 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,267 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,267 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:43,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:43,367 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:43,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:43,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:43,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:43,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:43,369 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:43,369 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:43,370 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:43,370 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:43,370 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,370 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:43,371 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:43,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:43,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:43,371 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:43,371 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,371 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:43,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:43,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:43,372 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:43,372 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:43,372 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:43,372 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:43,372 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:43,372 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:43,372 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:43,372 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:43,372 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:43,372 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:43,372 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:43,373 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:43,373 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:43,373 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:43,373 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,373 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:43,373 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:43,374 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,374 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:43,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:43,375 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,375 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:43,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:43,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:43,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:43,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:43,376 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:43,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@107d235f[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:43,376 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,376 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,376 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:43,376 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:43,377 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:43,377 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,377 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:43,377 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,377 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:43,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:43,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:43,377 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:43,377 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:43,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:43,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:43,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:43,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:43,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:43,379 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:43,379 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:43,379 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:43,379 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:43,379 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:43,380 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,380 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:43,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:43,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,380 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:43,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:43,381 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:43,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:43,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:43,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,382 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:43,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,383 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,383 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:43,383 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:43,383 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,383 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,384 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:43,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:43,384 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:43,384 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:43,384 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:43,384 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:43,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:43,384 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:43,384 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:43,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:43,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:43,484 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:43,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:43,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:43,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:43,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:43,487 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:43,487 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:43,487 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:43,487 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:43,487 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,487 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:43,488 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,488 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:43,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:43,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:43,488 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,488 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:43,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:43,488 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:43,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:43,489 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:43,489 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:43,489 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:43,489 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:43,489 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:43,489 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:43,489 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:43,489 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:43,489 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:43,489 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:43,489 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:43,489 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:43,489 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:43,490 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:43,490 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,490 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:43,490 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:43,490 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:43,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:43,491 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:43,491 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,491 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:43,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:43,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:43,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:43,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@645dee92[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:43,492 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:43,492 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,493 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,494 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:43,494 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:43,494 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:43,494 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,494 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:43,494 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:43,494 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,494 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:43,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:43,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:43,494 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:43,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:43,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:43,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:43,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:43,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:43,496 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:43,496 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:43,496 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:43,496 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:43,496 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:43,497 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,497 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,497 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:43,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:43,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:43,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:43,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:43,498 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:43,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:43,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,499 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:43,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,501 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,501 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:43,501 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:43,501 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:43,501 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:43,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:43,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:43,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:43,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:43,501 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:43,501 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:43,501 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,501 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,501 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:43,501 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:43,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:43,501 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,501 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:43,602 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:43,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:43,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:43,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:43,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:43,604 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:43,604 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:43,604 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:43,604 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:43,605 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,605 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:43,605 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,605 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:43,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:43,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:43,606 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,606 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:43,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:43,606 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:43,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:43,606 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:43,606 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:43,606 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:43,607 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:43,607 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:43,607 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:43,607 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:43,607 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:43,607 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:43,607 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:43,607 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:43,607 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:43,607 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:43,607 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,607 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:43,607 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:43,608 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:43,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:43,608 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:43,609 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:43,609 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:43,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:43,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:43,609 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:43,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c5c3260[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:43,609 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,610 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,610 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:43,610 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,610 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:43,610 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:43,610 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:43,611 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,611 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:43,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:43,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:43,611 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:43,611 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:43,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:43,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:43,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:43,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:43,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:43,613 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:43,613 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:43,613 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:43,613 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:43,613 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:43,614 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,614 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,614 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:43,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:43,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:43,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:43,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:43,614 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:43,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:43,615 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:43,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,617 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:43,617 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:43,617 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:43,617 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:43,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:43,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:43,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:43,617 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:43,617 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:43,617 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:43,617 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:43,617 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:43,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:43,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:43,617 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,618 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,618 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,618 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 969c768384097479d79029eb29913551 2023-04-20 12:54:43,702 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 969c768384097479d79029eb29913551 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-20 12:54:43,703 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=A 2023-04-20 12:54:43,703 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:43,703 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=B 2023-04-20 12:54:43,703 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:43,703 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=C 2023-04-20 12:54:43,703 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:43,718 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:43,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:43,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:43,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:43,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:43,729 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:43,729 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:43,729 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:43,729 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:43,729 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,729 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:43,730 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:43,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:43,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:43,730 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:43,731 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,731 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:43,731 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:43,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:43,732 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:43,732 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:43,732 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:43,732 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:43,732 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:43,732 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:43,734 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:43,734 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:43,734 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:43,734 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:43,734 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:43,734 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:43,734 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:43,734 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:43,734 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,734 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:43,734 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:43,739 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,739 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:43,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:43,739 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,739 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:43,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:43,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:43,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:43,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:43,740 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:43,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e767e2[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:43,740 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,741 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,741 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:43,741 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:43,741 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:43,742 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,742 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:43,743 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681995283722, max timestamp=1681995283741Num uniques 859; 2023-04-20 12:54:43,743 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 12:54:43,743 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681995283722, max timestamp=1681995283741Num uniques 859; 2023-04-20 12:54:43,743 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681995283722, max timestamp=1681995283741Num uniques 859; 2023-04-20 12:54:43,743 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 12:54:43,743 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 12:54:43,743 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:43,743 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:43,743 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:43,743 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:43,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:43,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:43,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:43,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:43,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:43,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:43,747 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:43,747 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:43,747 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:43,747 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:43,747 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:43,748 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,748 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:43,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:43,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,748 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:43,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:43,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:43,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:43,749 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:43,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,752 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:43,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,754 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,754 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:43,754 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:43,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:43,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:43,754 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:43,754 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:43,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:43,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:43,755 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:43,755 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:43,754 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,754 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:43,755 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:43,755 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,755 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,755 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:43,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:43,770 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:43,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40554 deadline: 1681995343767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,773 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:43,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40560 deadline: 1681995343771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,855 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:43,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:43,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:43,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:43,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:43,857 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:43,857 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:43,857 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:43,857 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:43,858 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,858 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:43,858 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:43,858 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:43,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:43,859 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,859 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:43,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:43,859 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:43,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:43,859 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:43,859 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:43,860 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:43,860 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:43,860 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:43,860 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:43,860 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:43,860 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:43,860 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:43,860 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:43,860 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:43,860 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:43,860 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:43,860 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:43,860 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,860 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:43,860 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:43,861 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,861 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:43,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:43,861 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,861 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:43,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:43,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:43,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:43,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:43,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e79cb9a[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:43,862 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:43,862 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,863 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,863 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:43,863 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:43,863 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:43,866 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,866 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:43,867 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,867 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:43,867 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:43,867 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:43,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:43,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:43,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:43,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:43,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:43,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:43,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:43,870 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:43,870 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:43,870 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:43,870 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:43,870 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:43,871 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,871 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:43,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:43,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,871 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:43,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:43,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:43,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,872 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:43,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:43,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,873 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,873 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:43,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40554 deadline: 1681995343871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,874 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:43,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40560 deadline: 1681995343874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:43,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,875 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,875 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:43,875 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:43,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:43,875 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:43,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:43,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:43,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:43,875 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:43,876 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,875 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:43,876 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,876 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,876 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:43,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:43,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:43,876 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:43,876 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:43,976 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:43,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:43,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:43,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:43,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:43,978 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:43,979 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:43,979 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:43,979 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:43,979 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,979 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:43,980 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:43,980 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:43,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:43,980 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,980 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:43,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:43,980 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:43,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:43,981 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:43,981 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:43,981 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:43,981 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:43,981 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:43,981 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:43,981 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:43,981 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:43,981 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:43,981 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:43,981 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:43,981 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:43,981 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:43,981 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:43,981 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,981 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:43,981 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:43,982 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:43,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:43,982 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:43,982 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,982 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:43,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:43,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:43,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:43,983 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:43,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@755b93b9[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:43,983 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,984 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,984 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:43,984 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:43,984 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:43,984 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,984 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:43,985 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,985 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:43,985 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:43,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:43,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:43,985 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:43,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:43,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:43,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:43,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:43,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:43,987 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:43,987 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:43,987 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:43,987 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:43,987 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:43,988 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,988 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:43,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:43,988 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:43,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:43,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:43,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:43,988 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:43,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,989 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:43,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:43,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,991 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,991 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:43,991 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:43,991 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:43,991 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:43,991 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,991 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:43,991 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:43,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:43,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:43,991 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:43,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:43,991 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:43,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:43,991 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:43,991 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:43,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:43,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:44,076 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:44,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40554 deadline: 1681995344074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,077 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:44,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40560 deadline: 1681995344075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,091 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:44,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:44,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:44,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:44,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:44,095 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:44,095 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:44,095 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:44,095 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:44,095 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:44,095 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:44,096 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:44,096 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:44,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:44,096 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,096 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:44,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:44,096 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:44,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:44,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:44,097 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:44,097 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:44,097 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:44,097 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:44,097 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:44,097 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:44,097 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:44,097 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:44,097 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:44,097 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:44,097 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:44,097 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:44,097 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:44,097 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:44,097 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,097 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:44,097 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:44,098 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,098 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:44,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:44,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:44,098 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:44,098 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:44,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:44,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:44,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:44,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:44,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:44,099 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:44,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78996040[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:44,099 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:44,100 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:44,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:44,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:44,100 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:44,100 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:44,100 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,100 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:44,100 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:44,101 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,101 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:44,101 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:44,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:44,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:44,101 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:44,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:44,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:44,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:44,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:44,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:44,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:44,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:44,102 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:44,102 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:44,102 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:44,102 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:44,102 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:44,103 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:44,103 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:44,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:44,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:44,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:44,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:44,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:44,103 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:44,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:44,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:44,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:44,104 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:44,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:44,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:44,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:44,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,104 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:44,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:44,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,106 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,106 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:44,106 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:44,106 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:44,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:44,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:44,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:44,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:44,106 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:44,106 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:44,106 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:44,106 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,106 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:44,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:44,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:44,106 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:44,106 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:44,106 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:44,110 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=288 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/787cf09f785e44aebb8713509bb90fcc 2023-04-20 12:54:44,206 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:44,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:44,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:44,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:44,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:44,210 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:44,210 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:44,210 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:44,210 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:44,210 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:44,210 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:44,211 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,211 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:44,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:44,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:44,211 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,211 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:44,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:44,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:44,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:44,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:44,212 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:44,212 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:44,212 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:44,212 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:44,212 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:44,212 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:44,212 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:44,212 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:44,212 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:44,212 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:44,212 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:44,212 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:44,212 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:44,212 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:44,212 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,212 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:44,212 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:44,213 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,213 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:44,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:44,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:44,213 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:44,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:44,214 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:44,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:44,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:44,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:44,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:44,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44fdc0c6[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:44,214 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:44,215 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:44,215 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:44,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:44,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:44,215 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:44,215 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:44,215 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:44,215 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,215 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:44,216 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:44,216 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:44,216 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,216 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:44,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:44,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:44,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:44,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:44,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:44,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:44,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:44,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:44,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:44,218 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:44,218 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:44,218 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:44,218 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:44,218 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:44,219 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:44,219 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:44,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:44,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:44,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:44,219 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:44,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:44,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:44,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:44,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:44,220 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:44,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:44,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:44,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:44,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:44,221 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:44,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:44,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,223 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,223 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:44,223 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:44,223 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,223 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:44,223 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:44,223 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:44,223 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:44,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:44,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:44,223 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:44,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:44,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:44,223 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:44,223 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:44,223 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:44,223 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:44,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:44,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:44,323 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:44,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:44,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:44,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:44,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:44,327 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:44,327 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:44,327 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:44,327 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:44,328 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:44,328 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:44,329 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:44,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:44,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:44,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:44,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:44,330 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:44,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:44,330 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:44,330 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,330 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:44,338 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:44,338 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:44,338 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:44,338 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:44,338 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:44,350 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:44,350 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:44,350 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:44,351 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:44,351 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:44,351 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:44,351 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:44,351 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:44,351 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,351 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:44,351 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:44,352 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:44,352 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:44,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:44,352 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:44,352 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:44,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:44,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:44,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:44,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:44,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:44,354 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:44,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7468e6d8[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:44,354 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:44,354 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:44,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:44,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:44,354 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:44,354 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:44,354 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:44,354 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,355 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:44,355 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,355 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:44,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:44,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:44,355 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:44,355 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:44,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:44,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:44,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:44,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:44,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:44,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:44,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:44,357 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:44,357 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:44,357 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:44,357 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:44,357 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:44,358 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:44,358 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:44,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:44,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:44,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:44,358 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:44,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:44,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:44,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:44,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:44,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:44,358 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:44,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:44,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:44,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:44,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,360 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:44,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:44,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,361 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:44,361 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:44,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:44,361 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:44,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:44,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:44,361 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:44,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:44,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:44,361 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:44,362 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:44,362 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,362 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:44,362 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:44,362 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:44,362 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:44,362 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:44,379 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:44,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40560 deadline: 1681995344378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,380 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:44,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40554 deadline: 1681995344379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,462 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:44,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:44,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:44,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:44,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:44,465 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:44,465 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:44,465 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:44,465 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:44,465 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:44,465 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:44,466 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:44,466 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:44,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:44,466 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,466 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:44,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:44,466 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:44,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:44,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:44,467 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:44,469 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:44,469 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:44,469 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:44,469 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:44,469 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:44,469 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:44,469 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:44,469 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:44,469 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:44,469 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:44,469 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:44,469 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:44,469 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:44,469 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,469 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:44,469 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:44,470 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,470 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:44,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:44,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:44,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:44,471 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:44,471 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:44,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:44,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:44,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:44,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:44,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d0775b7[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:44,472 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:44,472 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:44,472 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:44,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:44,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:44,473 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:44,473 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:44,473 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:44,473 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,473 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:44,473 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,473 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:44,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:44,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:44,473 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:44,474 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:44,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:44,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:44,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:44,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:44,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:44,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:44,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:44,475 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:44,475 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:44,475 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:44,475 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:44,475 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:44,476 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:44,476 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:44,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:44,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:44,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:44,476 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:44,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:44,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:44,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:44,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:44,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:44,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:44,477 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:44,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:44,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:44,478 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:44,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:44,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,479 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,479 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:44,479 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:44,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:44,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:44,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:44,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:44,479 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:44,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:44,479 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:44,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:44,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:44,479 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:44,480 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,480 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:44,480 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:44,480 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:44,480 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:44,480 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:44,519 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=288 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/1fae6d9869334991bc22b2b65d3e9628 2023-04-20 12:54:44,531 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=288 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/3f890e3f20cb43eeb1fc61a555fdb610 2023-04-20 12:54:44,536 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/787cf09f785e44aebb8713509bb90fcc as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/787cf09f785e44aebb8713509bb90fcc 2023-04-20 12:54:44,540 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/787cf09f785e44aebb8713509bb90fcc, entries=250, sequenceid=288, filesize=16.7 K 2023-04-20 12:54:44,540 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/1fae6d9869334991bc22b2b65d3e9628 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/1fae6d9869334991bc22b2b65d3e9628 2023-04-20 12:54:44,544 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/1fae6d9869334991bc22b2b65d3e9628, entries=150, sequenceid=288, filesize=11.9 K 2023-04-20 12:54:44,544 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/3f890e3f20cb43eeb1fc61a555fdb610 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/3f890e3f20cb43eeb1fc61a555fdb610 2023-04-20 12:54:44,548 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/3f890e3f20cb43eeb1fc61a555fdb610, entries=150, sequenceid=288, filesize=11.9 K 2023-04-20 12:54:44,548 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 969c768384097479d79029eb29913551 in 846ms, sequenceid=288, compaction requested=true 2023-04-20 12:54:44,548 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:44,548 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:44,548 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:54:44,549 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:54:44,549 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:44,549 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:54:44,549 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 41596 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:54:44,549 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36716 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:54:44,549 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): 969c768384097479d79029eb29913551/A is initiating minor compaction (all files) 2023-04-20 12:54:44,549 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): 969c768384097479d79029eb29913551/B is initiating minor compaction (all files) 2023-04-20 12:54:44,550 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 969c768384097479d79029eb29913551/A in TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:44,550 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 969c768384097479d79029eb29913551/B in TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:44,550 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/2735ca4481314a9e935dbd7c3529f5cb, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/d654be6e03524d82802a4211c680416f, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/1fae6d9869334991bc22b2b65d3e9628] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp, totalSize=35.9 K 2023-04-20 12:54:44,550 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/0daed446938d4efd8813719b59573423, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/7f9679f37bd1408099826e5df79a3cda, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/787cf09f785e44aebb8713509bb90fcc] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp, totalSize=40.6 K 2023-04-20 12:54:44,550 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 2735ca4481314a9e935dbd7c3529f5cb, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1681995281507 2023-04-20 12:54:44,550 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 0daed446938d4efd8813719b59573423, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1681995281507 2023-04-20 12:54:44,550 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting d654be6e03524d82802a4211c680416f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1681995281564 2023-04-20 12:54:44,550 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 7f9679f37bd1408099826e5df79a3cda, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1681995281564 2023-04-20 12:54:44,550 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 1fae6d9869334991bc22b2b65d3e9628, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1681995283701 2023-04-20 12:54:44,551 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 787cf09f785e44aebb8713509bb90fcc, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1681995283699 2023-04-20 12:54:44,557 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 969c768384097479d79029eb29913551#A#compaction#217 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:44,558 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): 969c768384097479d79029eb29913551#B#compaction#218 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:44,580 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:44,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:44,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:44,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:44,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:44,583 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:44,583 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:44,583 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:44,583 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:44,583 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:44,583 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:44,584 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:44,584 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:44,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:44,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:44,584 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,584 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:44,585 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:44,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:44,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:44,585 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:44,585 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:44,585 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:44,585 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:44,585 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:44,585 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:44,585 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:44,585 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:44,585 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:44,585 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:44,586 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 969c768384097479d79029eb29913551 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-20 12:54:44,586 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=A 2023-04-20 12:54:44,586 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:54:44,586 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=B 2023-04-20 12:54:44,586 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:54:44,586 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=C 2023-04-20 12:54:44,586 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:54:44,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 969c768384097479d79029eb29913551 2023-04-20 12:54:44,882 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:44,894 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:44,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40560 deadline: 1681995344892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,895 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:44,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40554 deadline: 1681995344893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:44,969 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/e9b3a9aef53f4a4aa0115103db81d446 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/e9b3a9aef53f4a4aa0115103db81d446 2023-04-20 12:54:44,971 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/fb7d5472239745a2948c6a428f08261f as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/fb7d5472239745a2948c6a428f08261f 2023-04-20 12:54:44,975 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 969c768384097479d79029eb29913551/B of 969c768384097479d79029eb29913551 into e9b3a9aef53f4a4aa0115103db81d446(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:54:44,975 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:44,975 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551., storeName=969c768384097479d79029eb29913551/B, priority=13, startTime=1681995284549; duration=0sec 2023-04-20 12:54:44,975 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:54:44,975 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:54:44,976 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36716 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:54:44,976 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): 969c768384097479d79029eb29913551/C is initiating minor compaction (all files) 2023-04-20 12:54:44,976 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 969c768384097479d79029eb29913551/C in TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:44,976 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/f1379ca7ad0f4f28a8ca4b59192cd475, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/00250837fd23488c908cec25c82873f4, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/3f890e3f20cb43eeb1fc61a555fdb610] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp, totalSize=35.9 K 2023-04-20 12:54:44,977 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting f1379ca7ad0f4f28a8ca4b59192cd475, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1681995281507 2023-04-20 12:54:44,977 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 00250837fd23488c908cec25c82873f4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1681995281564 2023-04-20 12:54:44,977 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 3f890e3f20cb43eeb1fc61a555fdb610, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1681995283701 2023-04-20 12:54:44,978 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 969c768384097479d79029eb29913551/A of 969c768384097479d79029eb29913551 into fb7d5472239745a2948c6a428f08261f(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:54:44,978 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:44,978 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551., storeName=969c768384097479d79029eb29913551/A, priority=13, startTime=1681995284548; duration=0sec 2023-04-20 12:54:44,978 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:44,983 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): 969c768384097479d79029eb29913551#C#compaction#220 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:44,990 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/fe16225f2fdc48b9a234d58f47783efe as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/fe16225f2fdc48b9a234d58f47783efe 2023-04-20 12:54:44,995 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 969c768384097479d79029eb29913551/C of 969c768384097479d79029eb29913551 into fe16225f2fdc48b9a234d58f47783efe(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:54:44,995 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:44,995 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551., storeName=969c768384097479d79029eb29913551/C, priority=13, startTime=1681995284549; duration=0sec 2023-04-20 12:54:44,995 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:44,995 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=323 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/405898cfc63f4079bc7a91ea24d1eecd 2023-04-20 12:54:44,996 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:44,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40560 deadline: 1681995344995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,001 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:45,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40554 deadline: 1681995344996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,010 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=323 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/b02c8cbd1b15444c852c01a1731b76f6 2023-04-20 12:54:45,199 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:45,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40560 deadline: 1681995345197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:45,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40554 deadline: 1681995345204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,430 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=323 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/6a9004bf92904432b67bb5cd65ce8c14 2023-04-20 12:54:45,434 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/405898cfc63f4079bc7a91ea24d1eecd as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/405898cfc63f4079bc7a91ea24d1eecd 2023-04-20 12:54:45,437 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/405898cfc63f4079bc7a91ea24d1eecd, entries=150, sequenceid=323, filesize=11.9 K 2023-04-20 12:54:45,438 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/b02c8cbd1b15444c852c01a1731b76f6 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/b02c8cbd1b15444c852c01a1731b76f6 2023-04-20 12:54:45,442 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/b02c8cbd1b15444c852c01a1731b76f6, entries=150, sequenceid=323, filesize=11.9 K 2023-04-20 12:54:45,442 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/6a9004bf92904432b67bb5cd65ce8c14 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/6a9004bf92904432b67bb5cd65ce8c14 2023-04-20 12:54:45,445 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/6a9004bf92904432b67bb5cd65ce8c14, entries=150, sequenceid=323, filesize=11.9 K 2023-04-20 12:54:45,446 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 969c768384097479d79029eb29913551 in 861ms, sequenceid=323, compaction requested=false 2023-04-20 12:54:45,446 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:45,446 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:45,446 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:45,446 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:45,446 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,446 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:45,446 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:45,447 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:45,447 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:45,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:45,448 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:45,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:45,448 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:45,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:45,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:45,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:45,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:45,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e2f9537[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:45,449 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:45,449 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:45,449 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:45,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:45,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:45,450 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:45,450 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:45,450 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:45,450 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,450 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:45,451 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,451 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:45,451 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:45,451 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:45,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:45,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:45,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:45,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:45,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:45,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:45,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:45,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:45,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:45,453 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:45,453 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:45,453 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:45,453 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:45,453 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:45,454 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:45,454 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:45,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:45,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:45,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:45,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:45,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:45,454 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:45,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:45,454 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:45,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:45,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:45,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:45,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:45,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:45,455 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:45,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:45,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,458 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:45,458 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,458 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:45,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:45,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:45,458 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:45,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:45,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:45,458 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:45,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:45,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:45,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:45,458 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:45,458 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:45,458 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,458 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:45,458 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:45,458 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:45,459 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:45,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 969c768384097479d79029eb29913551 2023-04-20 12:54:45,504 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 969c768384097479d79029eb29913551 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-20 12:54:45,504 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=A 2023-04-20 12:54:45,504 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:45,504 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=B 2023-04-20 12:54:45,504 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:45,504 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=C 2023-04-20 12:54:45,504 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:45,544 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:45,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40560 deadline: 1681995345542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,545 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:45,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40554 deadline: 1681995345543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,559 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:45,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:45,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:45,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:45,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:45,562 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:45,562 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:45,562 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:45,562 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:45,562 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:45,562 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:45,563 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:45,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:45,563 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:45,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:45,564 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,564 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:45,564 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:45,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:45,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:45,564 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:45,569 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:45,569 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:45,569 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:45,569 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:45,569 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:45,569 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:45,569 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:45,569 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:45,569 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:45,569 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:45,569 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:45,569 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:45,569 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:45,569 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,570 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:45,570 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:45,581 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:45,581 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:45,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:45,581 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:45,581 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:45,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:45,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:45,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:45,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:45,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:45,582 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:45,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44fe93c4[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:45,582 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:45,583 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:45,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:45,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:45,583 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:45,583 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:45,583 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:45,583 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,583 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:45,584 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:45,584 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,584 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:45,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:45,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:45,584 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:45,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:45,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:45,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:45,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:45,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:45,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:45,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:45,585 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:45,586 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:45,586 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:45,586 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:45,586 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:45,587 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:45,587 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:45,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:45,587 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:45,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:45,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:45,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:45,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:45,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:45,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:45,587 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:45,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:45,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:45,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:45,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:45,588 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:45,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:45,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,589 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,589 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:45,589 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:45,589 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:45,589 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:45,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:45,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:45,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:45,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:45,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:45,589 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,589 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:45,590 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:45,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:45,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:45,590 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:45,590 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:45,590 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:45,590 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:45,646 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:45,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40560 deadline: 1681995345645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,648 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:45,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40554 deadline: 1681995345646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,690 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:45,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:45,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:45,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:45,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:45,693 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:45,693 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:45,693 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:45,693 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:45,693 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:45,693 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:45,694 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,694 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:45,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:45,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:45,694 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,694 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:45,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:45,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:45,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:45,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:45,695 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:45,695 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:45,695 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:45,695 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:45,695 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:45,695 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:45,695 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:45,695 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:45,695 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:45,696 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:45,696 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:45,696 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:45,696 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:45,696 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:45,696 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,696 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:45,696 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:45,696 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,696 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:45,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:45,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:45,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:45,697 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:45,697 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:45,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:45,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:45,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:45,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:45,698 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:45,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7793e3a5[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:45,698 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:45,699 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:45,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:45,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:45,699 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:45,699 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:45,699 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,699 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:45,699 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:45,700 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,700 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:45,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:45,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:45,700 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:45,700 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:45,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:45,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:45,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:45,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:45,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:45,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:45,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:45,702 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:45,702 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:45,702 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:45,702 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:45,702 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:45,703 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:45,703 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:45,703 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:45,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:45,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:45,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:45,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:45,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:45,703 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:45,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:45,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:45,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:45,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:45,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:45,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:45,704 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:45,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:45,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,706 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,706 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:45,706 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:45,706 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:45,706 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:45,706 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:45,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:45,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:45,706 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:45,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:45,706 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:45,706 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:45,706 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:45,706 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:45,706 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:45,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:45,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:45,706 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:45,806 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:45,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:45,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:45,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:45,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:45,809 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:45,809 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:45,809 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:45,809 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:45,809 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:45,809 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:45,810 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:45,810 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:45,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:45,810 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,810 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:45,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:45,811 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:45,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:45,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:45,811 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:45,811 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:45,811 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:45,811 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:45,811 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:45,811 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:45,811 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:45,811 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:45,811 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:45,811 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:45,812 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:45,812 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:45,812 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:45,812 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:45,812 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,812 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:45,812 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:45,812 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,812 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:45,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:45,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:45,813 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:45,813 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:45,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:45,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:45,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:45,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:45,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:45,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11df8a2d[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:45,814 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:45,814 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:45,814 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:45,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:45,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:45,815 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:45,815 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:45,815 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:45,815 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,815 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:45,815 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,815 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:45,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:45,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:45,815 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:45,815 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:45,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:45,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:45,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:45,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:45,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:45,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:45,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:45,819 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:45,819 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:45,819 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:45,819 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:45,819 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:45,830 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:45,831 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:45,831 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:45,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:45,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:45,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:45,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:45,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:45,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:45,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:45,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:45,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:45,832 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:45,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:45,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:45,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,836 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:45,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:45,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,840 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,840 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:45,840 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:45,840 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,840 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:45,840 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:45,840 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:45,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:45,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:45,840 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:45,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:45,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:45,840 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:45,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:45,840 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:45,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:45,841 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:45,841 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:45,851 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:45,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40560 deadline: 1681995345848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,851 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:45,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40554 deadline: 1681995345849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,917 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=338 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/c7a2705510f04c15872fb2ab06bb7a45 2023-04-20 12:54:45,941 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:45,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:45,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:45,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:45,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:45,946 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:45,946 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:45,946 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:45,946 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:45,947 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:45,947 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:45,949 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:45,949 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:45,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:45,949 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,949 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:45,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:45,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:45,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:45,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:45,950 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:45,954 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:45,954 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:45,954 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:45,954 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:45,954 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:45,954 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:45,954 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:45,954 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:45,954 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:45,954 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:45,954 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:45,954 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:45,954 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:45,954 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,954 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:45,955 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:45,957 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,957 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:45,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:45,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:45,957 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:45,957 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:45,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:45,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:45,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:45,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:45,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:45,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34b330c[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:45,958 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:45,958 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:45,959 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:45,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:45,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:45,959 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:45,959 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:45,959 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:45,959 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,959 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:45,959 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,959 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:45,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:45,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:45,959 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:45,960 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:45,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:45,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:45,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:45,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:45,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:45,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:45,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:45,961 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:45,961 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:45,961 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:45,961 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:45,961 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:45,963 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:45,963 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:45,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:45,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:45,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:45,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:45,963 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:45,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:45,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:45,964 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:45,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:45,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:45,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:45,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:45,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:45,964 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:45,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:45,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,966 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,966 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:45,966 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:45,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:45,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:45,966 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:45,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:45,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:45,966 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:45,966 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:45,966 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:45,967 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:45,966 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:45,967 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:45,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:45,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:45,967 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:45,967 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:46,067 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:46,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:46,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:46,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:46,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:46,077 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:46,077 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:46,077 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:46,077 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:46,077 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:46,077 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:46,078 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:46,078 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:46,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:46,078 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,078 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:46,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:46,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:46,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:46,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:46,078 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:46,079 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:46,079 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:46,079 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:46,079 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:46,079 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:46,079 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:46,079 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:46,079 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:46,079 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:46,079 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:46,079 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:46,079 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:46,079 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:46,079 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,079 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:46,079 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:46,080 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,080 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:46,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:46,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:46,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:46,080 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:46,081 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:46,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:46,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:46,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:46,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:46,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@339bf012[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:46,081 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:46,081 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:46,082 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:46,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:46,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:46,082 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:46,082 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:46,082 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:46,082 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,082 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:46,083 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,084 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:46,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:46,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:46,084 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:46,084 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:46,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:46,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:46,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:46,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:46,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:46,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:46,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:46,088 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:46,088 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:46,088 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:46,088 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:46,088 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:46,089 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:46,089 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:46,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:46,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:46,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:46,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:46,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:46,089 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:46,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:46,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:46,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:46,091 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:46,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:46,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:46,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:46,092 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:46,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:46,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,096 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:46,096 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:46,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:46,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:46,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:46,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:46,096 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:46,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:46,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:46,097 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:46,097 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:46,098 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,098 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:46,098 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:46,098 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,098 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:46,098 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:46,098 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:46,153 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:46,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40560 deadline: 1681995346152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,156 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:46,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40554 deadline: 1681995346154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,197 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:46,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:46,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:46,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:46,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:46,200 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:46,200 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:46,200 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:46,200 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:46,200 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:46,200 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:46,201 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:46,201 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:46,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:46,201 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,201 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:46,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:46,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:46,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:46,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:46,202 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:46,202 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:46,202 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:46,202 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:46,202 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:46,202 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:46,202 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:46,202 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:46,202 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:46,202 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:46,202 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:46,202 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:46,202 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:46,202 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:46,202 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,202 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:46,202 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:46,203 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,203 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:46,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:46,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:46,203 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:46,203 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:46,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:46,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:46,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:46,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:46,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:46,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bfdb30f[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:46,204 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:46,204 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:46,205 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:46,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:46,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:46,205 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:46,205 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:46,205 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,205 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:46,205 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:46,206 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,206 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:46,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:46,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:46,206 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:46,206 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:46,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:46,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:46,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:46,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:46,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:46,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:46,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:46,208 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:46,208 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:46,208 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:46,208 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:46,208 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:46,209 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:46,209 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:46,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:46,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:46,209 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:46,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:46,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:46,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:46,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:46,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:46,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:46,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:46,210 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:46,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:46,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:46,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,211 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:46,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:46,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,212 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:46,212 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,212 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:46,212 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:46,212 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:46,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:46,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:46,212 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:46,212 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,212 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:46,212 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:46,212 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:46,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:46,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:46,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:46,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:46,213 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:46,213 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:46,313 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:46,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:46,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:46,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:46,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:46,316 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:46,316 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:46,316 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:46,316 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:46,316 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:46,316 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:46,317 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:46,317 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:46,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:46,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:46,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:46,318 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,318 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:46,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:46,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:46,318 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:46,318 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:46,318 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:46,318 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:46,318 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:46,318 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:46,326 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:46,326 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:46,326 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:46,326 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:46,326 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:46,326 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:46,326 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:46,326 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:46,326 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,326 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:46,326 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:46,331 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,331 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:46,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:46,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:46,331 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:46,331 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:46,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:46,331 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=338 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/c0052b67c89c48609a808267bebcad22 2023-04-20 12:54:46,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:46,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:46,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:46,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:46,334 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:46,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@361ec4bf[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:46,334 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:46,341 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:46,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:46,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:46,342 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:46,342 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,342 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:46,342 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:46,342 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:46,347 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,347 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:46,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:46,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:46,347 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:46,348 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:46,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:46,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:46,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:46,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:46,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:46,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:46,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:46,349 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:46,349 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:46,349 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:46,349 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:46,349 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:46,352 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=338 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/a874bad3e90a468887728c67b5ed7808 2023-04-20 12:54:46,357 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:46,357 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:46,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:46,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:46,357 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:46,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:46,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:46,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:46,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:46,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:46,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:46,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:46,358 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:46,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:46,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:46,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,359 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:46,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:46,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,361 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/c7a2705510f04c15872fb2ab06bb7a45 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/c7a2705510f04c15872fb2ab06bb7a45 2023-04-20 12:54:46,365 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:46,365 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:46,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:46,365 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:46,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:46,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:46,366 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:46,365 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:46,365 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:46,366 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:46,365 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:46,366 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,366 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:46,366 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:46,366 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:46,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:46,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:46,369 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/c7a2705510f04c15872fb2ab06bb7a45, entries=150, sequenceid=338, filesize=11.9 K 2023-04-20 12:54:46,371 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/c0052b67c89c48609a808267bebcad22 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/c0052b67c89c48609a808267bebcad22 2023-04-20 12:54:46,374 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/c0052b67c89c48609a808267bebcad22, entries=150, sequenceid=338, filesize=11.9 K 2023-04-20 12:54:46,375 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/a874bad3e90a468887728c67b5ed7808 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/a874bad3e90a468887728c67b5ed7808 2023-04-20 12:54:46,378 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/a874bad3e90a468887728c67b5ed7808, entries=150, sequenceid=338, filesize=11.9 K 2023-04-20 12:54:46,379 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 969c768384097479d79029eb29913551 in 875ms, sequenceid=338, compaction requested=true 2023-04-20 12:54:46,379 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:46,379 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:46,379 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:54:46,379 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:54:46,379 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:46,379 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:54:46,380 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36968 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:54:46,380 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): 969c768384097479d79029eb29913551/A is initiating minor compaction (all files) 2023-04-20 12:54:46,380 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36968 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:54:46,380 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 969c768384097479d79029eb29913551/A in TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:46,380 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): 969c768384097479d79029eb29913551/B is initiating minor compaction (all files) 2023-04-20 12:54:46,380 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/fb7d5472239745a2948c6a428f08261f, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/405898cfc63f4079bc7a91ea24d1eecd, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/c7a2705510f04c15872fb2ab06bb7a45] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp, totalSize=36.1 K 2023-04-20 12:54:46,380 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 969c768384097479d79029eb29913551/B in TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:46,380 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/e9b3a9aef53f4a4aa0115103db81d446, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/b02c8cbd1b15444c852c01a1731b76f6, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/c0052b67c89c48609a808267bebcad22] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp, totalSize=36.1 K 2023-04-20 12:54:46,381 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting e9b3a9aef53f4a4aa0115103db81d446, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1681995283701 2023-04-20 12:54:46,381 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting fb7d5472239745a2948c6a428f08261f, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1681995283701 2023-04-20 12:54:46,381 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting b02c8cbd1b15444c852c01a1731b76f6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=323, earliestPutTs=1681995283756 2023-04-20 12:54:46,381 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 405898cfc63f4079bc7a91ea24d1eecd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=323, earliestPutTs=1681995283756 2023-04-20 12:54:46,381 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting c0052b67c89c48609a808267bebcad22, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1681995284884 2023-04-20 12:54:46,381 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting c7a2705510f04c15872fb2ab06bb7a45, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1681995284884 2023-04-20 12:54:46,387 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): 969c768384097479d79029eb29913551#B#compaction#226 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:46,389 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 969c768384097479d79029eb29913551#A#compaction#227 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:46,411 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/11de23e5dcbb4ed0bebc97267f64ec43 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/11de23e5dcbb4ed0bebc97267f64ec43 2023-04-20 12:54:46,416 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 969c768384097479d79029eb29913551/B of 969c768384097479d79029eb29913551 into 11de23e5dcbb4ed0bebc97267f64ec43(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:54:46,416 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:46,416 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551., storeName=969c768384097479d79029eb29913551/B, priority=13, startTime=1681995286379; duration=0sec 2023-04-20 12:54:46,416 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:54:46,416 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:54:46,417 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36968 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:54:46,417 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): 969c768384097479d79029eb29913551/C is initiating minor compaction (all files) 2023-04-20 12:54:46,417 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 969c768384097479d79029eb29913551/C in TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:46,417 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/fe16225f2fdc48b9a234d58f47783efe, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/6a9004bf92904432b67bb5cd65ce8c14, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/a874bad3e90a468887728c67b5ed7808] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp, totalSize=36.1 K 2023-04-20 12:54:46,418 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting fe16225f2fdc48b9a234d58f47783efe, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=288, earliestPutTs=1681995283701 2023-04-20 12:54:46,418 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 6a9004bf92904432b67bb5cd65ce8c14, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=323, earliestPutTs=1681995283756 2023-04-20 12:54:46,418 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting a874bad3e90a468887728c67b5ed7808, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1681995284884 2023-04-20 12:54:46,424 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): 969c768384097479d79029eb29913551#C#compaction#228 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:46,467 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:46,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:46,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:46,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:46,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:46,470 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:46,470 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:46,470 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:46,470 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:46,471 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:46,471 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:46,472 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:46,472 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:46,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:46,472 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,472 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:46,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:46,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:46,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:46,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:46,473 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:46,474 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:46,474 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:46,474 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:46,474 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:46,474 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:46,474 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:46,474 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:46,474 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:46,474 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 969c768384097479d79029eb29913551 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-20 12:54:46,475 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=A 2023-04-20 12:54:46,475 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:46,475 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=B 2023-04-20 12:54:46,475 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:46,475 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=C 2023-04-20 12:54:46,475 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:46,499 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=359 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/d3a6e3f8afbb403399b06633fb598d27 2023-04-20 12:54:46,520 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=359 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/37a874107a7d49c392ec4f162e7233a8 2023-04-20 12:54:46,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 969c768384097479d79029eb29913551 2023-04-20 12:54:46,659 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:46,673 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:46,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40560 deadline: 1681995346671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,674 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:46,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40554 deadline: 1681995346672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,689 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:46,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40616 deadline: 1681995346687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,690 DEBUG [Thread-1494] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8159 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551., hostname=jenkins-hbase20.apache.org,36277,1681995213947, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-20 12:54:46,706 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:46,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40570 deadline: 1681995346705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,707 DEBUG [Thread-1488] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8174 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551., hostname=jenkins-hbase20.apache.org,36277,1681995213947, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-20 12:54:46,713 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:46,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40568 deadline: 1681995346712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,717 DEBUG [Thread-1492] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8183 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551., hostname=jenkins-hbase20.apache.org,36277,1681995213947, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-20 12:54:46,775 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:46,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40560 deadline: 1681995346774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,775 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:46,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40554 deadline: 1681995346774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,816 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/7dab9f1c1b404e38a3c8dac8a9e3cf87 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/7dab9f1c1b404e38a3c8dac8a9e3cf87 2023-04-20 12:54:46,820 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 969c768384097479d79029eb29913551/A of 969c768384097479d79029eb29913551 into 7dab9f1c1b404e38a3c8dac8a9e3cf87(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:54:46,820 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:46,820 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551., storeName=969c768384097479d79029eb29913551/A, priority=13, startTime=1681995286379; duration=0sec 2023-04-20 12:54:46,820 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:46,835 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/3f68a3f4c12446db9846b95a839e35b8 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/3f68a3f4c12446db9846b95a839e35b8 2023-04-20 12:54:46,839 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 969c768384097479d79029eb29913551/C of 969c768384097479d79029eb29913551 into 3f68a3f4c12446db9846b95a839e35b8(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:54:46,839 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:46,839 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551., storeName=969c768384097479d79029eb29913551/C, priority=13, startTime=1681995286379; duration=0sec 2023-04-20 12:54:46,839 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:46,937 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=359 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/5ecc31bcb3c64e3c94eacf34ee997ebe 2023-04-20 12:54:46,941 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/d3a6e3f8afbb403399b06633fb598d27 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/d3a6e3f8afbb403399b06633fb598d27 2023-04-20 12:54:46,944 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/d3a6e3f8afbb403399b06633fb598d27, entries=150, sequenceid=359, filesize=11.9 K 2023-04-20 12:54:46,945 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/37a874107a7d49c392ec4f162e7233a8 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/37a874107a7d49c392ec4f162e7233a8 2023-04-20 12:54:46,948 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/37a874107a7d49c392ec4f162e7233a8, entries=150, sequenceid=359, filesize=11.9 K 2023-04-20 12:54:46,949 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/5ecc31bcb3c64e3c94eacf34ee997ebe as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/5ecc31bcb3c64e3c94eacf34ee997ebe 2023-04-20 12:54:46,956 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/5ecc31bcb3c64e3c94eacf34ee997ebe, entries=150, sequenceid=359, filesize=11.9 K 2023-04-20 12:54:46,957 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 969c768384097479d79029eb29913551 in 483ms, sequenceid=359, compaction requested=false 2023-04-20 12:54:46,957 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:46,957 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:46,957 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:46,957 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:46,957 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,957 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:46,957 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:46,958 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,958 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:46,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:46,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:46,958 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:46,958 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:46,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:46,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:46,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:46,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:46,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:46,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c175223[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:46,959 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:46,960 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:46,960 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:46,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:46,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:46,960 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,960 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:46,960 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:46,960 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:46,960 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:46,961 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:46,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:46,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:46,962 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:46,962 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:46,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:46,962 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:46,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:46,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:46,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:46,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:46,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:46,963 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:46,963 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:46,963 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:46,963 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:46,963 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:46,964 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:46,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:46,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:46,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:46,964 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:46,964 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:46,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:46,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:46,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:46,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:46,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:46,965 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:46,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:46,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:46,966 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:46,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:46,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:46,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,967 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,967 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:46,967 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:46,968 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:46,968 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:46,968 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:46,968 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:46,968 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:46,968 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:46,968 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:46,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:46,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:46,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:46,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:46,968 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:46,968 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:46,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:46,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:46,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:46,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 969c768384097479d79029eb29913551 2023-04-20 12:54:46,978 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 969c768384097479d79029eb29913551 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-20 12:54:46,978 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=A 2023-04-20 12:54:46,978 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:46,978 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=B 2023-04-20 12:54:46,978 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:46,978 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=C 2023-04-20 12:54:46,978 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:47,007 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:47,007 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:47,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40560 deadline: 1681995347004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40554 deadline: 1681995347004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,024 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/3a7a3e8c0b2940a6896586d35b481942 2023-04-20 12:54:47,047 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/1ad6e3d840c14525b6423b948ed3efbf 2023-04-20 12:54:47,068 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:47,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:47,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:47,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:47,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:47,071 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:47,071 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:47,071 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:47,071 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:47,071 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,071 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:47,072 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:47,072 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:47,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:47,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:47,073 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,073 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:47,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:47,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:47,073 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:47,074 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:47,074 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:47,074 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:47,074 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:47,074 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:47,080 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:47,080 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:47,080 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:47,080 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:47,080 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:47,080 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:47,080 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:47,080 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:47,080 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,080 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:47,080 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:47,097 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,097 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:47,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:47,098 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,098 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:47,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:47,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:47,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:47,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:47,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:47,099 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:47,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e024d3[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:47,099 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,099 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,100 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:47,100 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:47,100 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:47,100 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,100 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:47,100 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,100 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:47,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:47,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:47,100 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:47,100 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:47,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:47,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:47,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:47,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:47,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:47,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:47,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:47,102 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:47,102 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:47,102 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:47,102 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:47,102 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:47,103 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,103 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,103 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:47,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:47,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:47,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,103 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:47,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:47,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:47,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:47,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:47,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:47,104 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:47,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:47,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,106 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:47,106 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:47,106 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:47,106 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:47,106 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:47,106 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:47,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:47,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:47,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:47,106 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:47,106 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,106 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,106 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:47,106 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,106 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:47,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:47,106 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:47,110 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:47,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40560 deadline: 1681995347108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,113 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:47,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40554 deadline: 1681995347111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,206 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:47,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:47,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:47,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:47,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:47,218 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:47,218 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:47,218 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:47,218 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:47,218 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,218 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:47,219 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:47,219 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:47,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:47,219 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,219 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:47,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:47,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:47,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:47,220 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:47,220 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:47,220 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:47,220 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:47,220 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:47,220 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:47,226 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:47,226 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:47,226 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:47,226 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:47,226 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:47,226 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:47,226 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:47,226 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:47,226 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,226 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:47,226 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:47,227 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:47,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:47,227 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:47,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:47,228 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,228 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:47,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:47,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:47,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:47,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36b39a0b[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:47,229 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:47,229 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,230 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,230 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:47,230 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:47,230 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:47,230 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,230 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:47,233 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:47,233 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:47,233 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:47,233 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:47,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:47,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:47,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:47,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:47,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:47,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:47,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:47,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:47,236 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:47,236 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:47,236 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:47,236 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:47,236 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:47,237 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,237 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,237 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:47,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:47,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:47,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:47,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:47,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:47,238 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:47,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:47,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:47,239 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:47,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:47,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,241 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:47,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:47,241 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:47,241 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,241 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:47,241 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:47,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:47,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:47,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:47,241 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:47,241 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:47,241 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,241 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,241 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:47,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:47,242 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:47,242 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:47,242 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,313 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:47,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40560 deadline: 1681995347312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,317 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:47,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40554 deadline: 1681995347315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,342 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:47,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:47,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:47,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:47,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:47,344 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:47,344 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:47,344 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:47,345 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:47,345 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,345 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:47,346 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:47,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:47,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:47,346 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:47,346 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,347 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:47,347 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:47,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:47,347 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:47,347 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:47,347 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:47,347 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:47,347 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:47,347 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:47,348 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:47,348 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:47,348 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:47,348 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:47,348 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:47,348 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:47,348 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:47,348 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:47,348 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,348 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:47,348 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:47,349 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,349 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:47,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:47,349 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,349 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:47,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:47,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:47,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:47,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:47,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:47,351 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:47,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76c88b5e[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:47,351 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,351 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,351 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:47,351 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:47,351 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:47,352 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,352 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:47,352 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,352 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:47,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:47,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:47,352 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:47,352 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:47,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:47,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:47,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:47,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:47,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:47,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:47,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:47,354 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:47,354 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:47,354 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:47,354 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:47,354 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:47,356 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,356 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:47,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:47,356 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:47,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:47,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:47,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:47,356 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:47,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:47,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:47,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,357 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:47,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:47,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,365 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,365 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:47,365 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:47,365 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,365 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:47,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:47,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:47,366 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:47,366 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:47,365 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:47,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:47,366 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:47,365 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:47,365 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:47,366 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,366 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:47,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:47,456 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/856a9d3640994a898b7e528c560f38e2 2023-04-20 12:54:47,462 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/3a7a3e8c0b2940a6896586d35b481942 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/3a7a3e8c0b2940a6896586d35b481942 2023-04-20 12:54:47,466 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:47,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:47,467 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/3a7a3e8c0b2940a6896586d35b481942, entries=200, sequenceid=374, filesize=14.3 K 2023-04-20 12:54:47,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:47,469 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/1ad6e3d840c14525b6423b948ed3efbf as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/1ad6e3d840c14525b6423b948ed3efbf 2023-04-20 12:54:47,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:47,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:47,470 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:47,470 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:47,470 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:47,470 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:47,470 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,470 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:47,471 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:47,471 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:47,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:47,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:47,472 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,472 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:47,472 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:47,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:47,472 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:47,472 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:47,472 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:47,473 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:47,473 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:47,473 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:47,473 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:47,473 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:47,473 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:47,473 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:47,473 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:47,473 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:47,473 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:47,473 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:47,473 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,473 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:47,473 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:47,474 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/1ad6e3d840c14525b6423b948ed3efbf, entries=150, sequenceid=374, filesize=11.9 K 2023-04-20 12:54:47,475 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,475 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,475 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:47,475 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:47,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:47,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:47,476 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/856a9d3640994a898b7e528c560f38e2 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/856a9d3640994a898b7e528c560f38e2 2023-04-20 12:54:47,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:47,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:47,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:47,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:47,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@209859ee[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:47,477 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:47,477 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,477 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,478 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:47,478 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:47,478 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,478 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:47,478 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:47,479 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,479 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:47,479 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:47,479 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:47,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:47,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:47,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:47,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:47,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:47,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:47,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:47,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,482 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/856a9d3640994a898b7e528c560f38e2, entries=150, sequenceid=374, filesize=11.9 K 2023-04-20 12:54:47,482 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 969c768384097479d79029eb29913551 in 504ms, sequenceid=374, compaction requested=true 2023-04-20 12:54:47,482 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:47,483 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:54:47,483 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:54:47,483 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:54:47,483 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:54:47,483 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:54:47,484 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39510 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:54:47,484 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): 969c768384097479d79029eb29913551/A is initiating minor compaction (all files) 2023-04-20 12:54:47,484 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 969c768384097479d79029eb29913551/A in TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:47,484 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/7dab9f1c1b404e38a3c8dac8a9e3cf87, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/d3a6e3f8afbb403399b06633fb598d27, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/3a7a3e8c0b2940a6896586d35b481942] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp, totalSize=38.6 K 2023-04-20 12:54:47,485 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 7dab9f1c1b404e38a3c8dac8a9e3cf87, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1681995284884 2023-04-20 12:54:47,485 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:54:47,485 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): 969c768384097479d79029eb29913551/B is initiating minor compaction (all files) 2023-04-20 12:54:47,485 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 969c768384097479d79029eb29913551/B in TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:47,485 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/11de23e5dcbb4ed0bebc97267f64ec43, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/37a874107a7d49c392ec4f162e7233a8, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/1ad6e3d840c14525b6423b948ed3efbf] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp, totalSize=36.2 K 2023-04-20 12:54:47,486 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting d3a6e3f8afbb403399b06633fb598d27, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1681995285535 2023-04-20 12:54:47,486 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 11de23e5dcbb4ed0bebc97267f64ec43, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1681995284884 2023-04-20 12:54:47,486 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 3a7a3e8c0b2940a6896586d35b481942, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1681995286663 2023-04-20 12:54:47,486 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 37a874107a7d49c392ec4f162e7233a8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1681995285535 2023-04-20 12:54:47,487 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 1ad6e3d840c14525b6423b948ed3efbf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1681995286663 2023-04-20 12:54:47,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:47,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:47,487 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:47,487 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:47,487 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:47,487 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:47,487 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:47,490 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,490 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:47,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,490 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:47,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:47,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:47,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:47,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:47,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:47,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:47,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:47,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:47,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:47,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,495 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): 969c768384097479d79029eb29913551#A#compaction#235 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:47,498 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,499 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,500 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 969c768384097479d79029eb29913551#B#compaction#236 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:47,500 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,500 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:47,500 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:47,501 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,501 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:47,501 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:47,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:47,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:47,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:47,501 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,501 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:47,501 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:47,501 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:47,501 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,501 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:47,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:47,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:47,534 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/b1e672be0ab24c2e932b7d0e02e0133a as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/b1e672be0ab24c2e932b7d0e02e0133a 2023-04-20 12:54:47,543 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 969c768384097479d79029eb29913551/A of 969c768384097479d79029eb29913551 into b1e672be0ab24c2e932b7d0e02e0133a(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:54:47,543 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:47,543 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551., storeName=969c768384097479d79029eb29913551/A, priority=13, startTime=1681995287483; duration=0sec 2023-04-20 12:54:47,543 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:54:47,543 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:54:47,545 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:54:47,545 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): 969c768384097479d79029eb29913551/C is initiating minor compaction (all files) 2023-04-20 12:54:47,545 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 969c768384097479d79029eb29913551/C in TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:47,545 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/3f68a3f4c12446db9846b95a839e35b8, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/5ecc31bcb3c64e3c94eacf34ee997ebe, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/856a9d3640994a898b7e528c560f38e2] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp, totalSize=36.2 K 2023-04-20 12:54:47,546 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 3f68a3f4c12446db9846b95a839e35b8, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1681995284884 2023-04-20 12:54:47,551 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 5ecc31bcb3c64e3c94eacf34ee997ebe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1681995285535 2023-04-20 12:54:47,551 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 856a9d3640994a898b7e528c560f38e2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1681995286663 2023-04-20 12:54:47,556 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/8c04bf595cb54203a35d56d7b6585511 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/8c04bf595cb54203a35d56d7b6585511 2023-04-20 12:54:47,563 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): 969c768384097479d79029eb29913551#C#compaction#237 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:47,565 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 969c768384097479d79029eb29913551/B of 969c768384097479d79029eb29913551 into 8c04bf595cb54203a35d56d7b6585511(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:54:47,565 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:47,565 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551., storeName=969c768384097479d79029eb29913551/B, priority=13, startTime=1681995287483; duration=0sec 2023-04-20 12:54:47,565 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:47,597 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/bfae4ce02339452faa94eaa462bc031c as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/bfae4ce02339452faa94eaa462bc031c 2023-04-20 12:54:47,601 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:47,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:47,603 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 969c768384097479d79029eb29913551/C of 969c768384097479d79029eb29913551 into bfae4ce02339452faa94eaa462bc031c(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:54:47,603 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:47,603 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551., storeName=969c768384097479d79029eb29913551/C, priority=13, startTime=1681995287483; duration=0sec 2023-04-20 12:54:47,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:47,603 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:47,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:47,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:47,605 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:47,605 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:47,605 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:47,605 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:47,605 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,605 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:47,606 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,606 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:47,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:47,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:47,606 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,606 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:47,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:47,607 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:47,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:47,608 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:47,610 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:47,610 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:47,610 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:47,610 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:47,610 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:47,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 969c768384097479d79029eb29913551 2023-04-20 12:54:47,616 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 969c768384097479d79029eb29913551 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-20 12:54:47,616 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=A 2023-04-20 12:54:47,616 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:47,616 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=B 2023-04-20 12:54:47,616 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:47,616 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=C 2023-04-20 12:54:47,617 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:47,621 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:47,621 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:47,621 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:47,621 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:47,622 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:47,622 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:47,622 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:47,622 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:47,622 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,622 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:47,622 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:47,623 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,624 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,624 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:47,624 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:47,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:47,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:47,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:47,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:47,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:47,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:47,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21df2792[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:47,625 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:47,625 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,626 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,626 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:47,626 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:47,626 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:47,627 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,627 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:47,627 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,627 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:47,627 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:47,627 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:47,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:47,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:47,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:47,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:47,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:47,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:47,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:47,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:47,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:47,629 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:47,629 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:47,629 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:47,629 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:47,629 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:47,631 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,631 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:47,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:47,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,631 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:47,631 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:47,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:47,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:47,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:47,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:47,632 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:47,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,634 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:47,634 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:47,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:47,634 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:47,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:47,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:47,634 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:47,634 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:47,634 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:47,634 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,634 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,634 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,634 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:47,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:47,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:47,634 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:47,635 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:47,635 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:47,650 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:47,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40554 deadline: 1681995347648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,650 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:47,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40560 deadline: 1681995347649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,735 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:47,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:47,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:47,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:47,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:47,738 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:47,738 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:47,738 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:47,738 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:47,738 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,738 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:47,739 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:47,739 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:47,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:47,739 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,739 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:47,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:47,740 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:47,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:47,740 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:47,741 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:47,741 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:47,741 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:47,741 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:47,741 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:47,741 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:47,741 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:47,741 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:47,741 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:47,741 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:47,741 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:47,741 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:47,741 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:47,741 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,741 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:47,741 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:47,743 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,743 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:47,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:47,743 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,743 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:47,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:47,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:47,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:47,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:47,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:47,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b969260[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:47,745 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:47,745 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,746 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,746 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:47,746 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:47,746 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:47,747 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,747 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:47,747 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:47,747 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:47,747 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:47,747 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:47,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:47,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:47,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:47,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:47,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:47,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:47,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:47,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:47,750 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:47,750 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:47,750 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:47,750 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:47,750 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:47,751 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:47,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40560 deadline: 1681995347751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,751 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,751 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:47,751 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40554 deadline: 1681995347751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:47,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:47,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,752 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:47,752 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:47,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:47,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:47,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:47,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:47,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:47,753 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:47,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:47,755 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,755 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:47,755 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:47,755 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:47,755 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:47,755 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:47,755 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:47,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:47,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:47,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:47,755 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,755 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,756 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:47,756 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:47,755 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:47,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:47,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:47,756 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,856 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:47,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:47,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:47,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:47,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:47,859 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:47,859 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:47,859 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:47,859 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:47,860 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,860 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:47,860 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:47,860 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:47,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:47,862 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:47,862 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:47,862 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:47,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:47,862 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:47,862 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:47,862 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:47,862 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:47,862 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:47,862 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:47,870 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:47,870 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:47,870 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:47,870 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:47,870 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:47,870 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:47,870 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:47,870 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:47,870 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,870 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:47,870 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:47,873 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,873 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:47,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:47,874 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,874 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:47,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:47,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:47,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:47,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:47,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:47,876 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:47,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39b66a37[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:47,876 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,877 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,877 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:47,877 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:47,877 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:47,877 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,877 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:47,878 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:47,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:47,878 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:47,878 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:47,878 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:47,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:47,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:47,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:47,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:47,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:47,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:47,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:47,880 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:47,880 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:47,880 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:47,880 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:47,880 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:47,881 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,881 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,881 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:47,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:47,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:47,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:47,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:47,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:47,881 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:47,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:47,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:47,882 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:47,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:47,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,884 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:47,884 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:47,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:47,884 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:47,885 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:47,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:47,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:47,884 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:47,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:47,885 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:47,885 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,885 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,885 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:47,885 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,885 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:47,885 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:47,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:47,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:47,954 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:47,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40560 deadline: 1681995347953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,955 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:47,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40554 deadline: 1681995347953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,985 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:47,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:47,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:47,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:47,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:47,988 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:47,988 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:47,988 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:47,988 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:47,990 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,990 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:47,991 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:47,991 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:47,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:47,991 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:47,991 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:47,992 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:47,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:47,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:47,992 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:47,995 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:47,995 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:47,995 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:47,995 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:47,995 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:47,996 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:47,996 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:47,996 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:47,996 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:47,996 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:47,996 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:47,996 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:47,996 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:47,996 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:47,996 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:47,996 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:48,006 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,006 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:48,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:48,006 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,006 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:48,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:48,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:48,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:48,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:48,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:48,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b0313c6[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:48,007 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:48,007 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,008 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,008 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:48,008 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:48,008 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:48,009 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,009 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:48,009 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,009 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:48,009 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:48,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:48,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:48,009 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:48,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:48,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:48,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:48,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:48,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:48,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:48,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:48,012 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:48,012 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:48,012 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:48,012 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:48,012 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:48,013 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:48,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:48,014 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,014 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:48,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:48,014 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:48,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:48,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:48,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:48,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:48,015 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:48,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:48,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,018 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,018 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:48,018 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:48,018 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:48,018 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:48,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:48,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:48,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:48,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:48,018 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,018 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,018 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,018 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,018 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:48,018 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:48,018 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:48,018 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:48,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:48,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:48,037 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=397 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/29d0ee9899d34e0099d4a7746913f829 2023-04-20 12:54:48,046 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=397 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/6752be952a5b4e309061525c71d0a106 2023-04-20 12:54:48,063 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=397 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/0a77acd8cdd1474489e5416eaf89fd5d 2023-04-20 12:54:48,071 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/29d0ee9899d34e0099d4a7746913f829 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/29d0ee9899d34e0099d4a7746913f829 2023-04-20 12:54:48,075 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/29d0ee9899d34e0099d4a7746913f829, entries=150, sequenceid=397, filesize=11.9 K 2023-04-20 12:54:48,076 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/6752be952a5b4e309061525c71d0a106 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/6752be952a5b4e309061525c71d0a106 2023-04-20 12:54:48,082 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/6752be952a5b4e309061525c71d0a106, entries=150, sequenceid=397, filesize=11.9 K 2023-04-20 12:54:48,083 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/0a77acd8cdd1474489e5416eaf89fd5d as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/0a77acd8cdd1474489e5416eaf89fd5d 2023-04-20 12:54:48,088 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/0a77acd8cdd1474489e5416eaf89fd5d, entries=150, sequenceid=397, filesize=11.9 K 2023-04-20 12:54:48,089 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 969c768384097479d79029eb29913551 in 473ms, sequenceid=397, compaction requested=false 2023-04-20 12:54:48,089 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:48,119 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:48,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:48,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:48,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:48,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:48,130 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:48,130 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:48,130 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:48,130 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:48,131 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,131 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:48,131 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:48,131 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:48,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:48,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:48,132 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,132 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,132 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:48,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:48,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:48,133 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:48,138 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:48,138 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:48,138 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:48,138 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:48,138 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:48,139 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:48,139 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:48,139 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:48,140 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:48,140 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 969c768384097479d79029eb29913551 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-20 12:54:48,142 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=A 2023-04-20 12:54:48,142 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:48,142 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=B 2023-04-20 12:54:48,142 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:48,142 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=C 2023-04-20 12:54:48,142 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:48,170 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=409 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/27e82770a2fb4bdba87e4f2a551cb15c 2023-04-20 12:54:48,187 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=409 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/577d6977dc2f47eea5af5a7d98974f5e 2023-04-20 12:54:48,205 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=409 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/b731c58d0de249258e2a7dca947a08a1 2023-04-20 12:54:48,210 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/27e82770a2fb4bdba87e4f2a551cb15c as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/27e82770a2fb4bdba87e4f2a551cb15c 2023-04-20 12:54:48,215 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/27e82770a2fb4bdba87e4f2a551cb15c, entries=150, sequenceid=409, filesize=11.9 K 2023-04-20 12:54:48,218 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/577d6977dc2f47eea5af5a7d98974f5e as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/577d6977dc2f47eea5af5a7d98974f5e 2023-04-20 12:54:48,223 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/577d6977dc2f47eea5af5a7d98974f5e, entries=150, sequenceid=409, filesize=11.9 K 2023-04-20 12:54:48,223 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/b731c58d0de249258e2a7dca947a08a1 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/b731c58d0de249258e2a7dca947a08a1 2023-04-20 12:54:48,227 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/b731c58d0de249258e2a7dca947a08a1, entries=150, sequenceid=409, filesize=11.9 K 2023-04-20 12:54:48,228 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for 969c768384097479d79029eb29913551 in 88ms, sequenceid=409, compaction requested=true 2023-04-20 12:54:48,228 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:48,228 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:48,228 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:48,229 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:48,229 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,229 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:48,229 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:48,231 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:48,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:48,231 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:48,232 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,232 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:48,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:48,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:48,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:48,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:48,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b85c83[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:48,233 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:48,233 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,234 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,234 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:48,234 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:48,234 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:48,234 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,234 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:48,235 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:48,235 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:48,235 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:48,235 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:48,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:48,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:48,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:48,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:48,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:48,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:48,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:48,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:48,238 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:48,238 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:48,238 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:48,238 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:48,238 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:48,239 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,239 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:48,239 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:48,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:48,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:48,240 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:48,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:48,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:48,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:48,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:48,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,242 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:48,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:48,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,244 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,244 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:48,244 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:48,244 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:48,244 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,244 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:48,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:48,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:48,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:48,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:48,244 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:48,245 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:48,244 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:48,245 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:48,245 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:48,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:48,245 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 969c768384097479d79029eb29913551 2023-04-20 12:54:48,263 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 969c768384097479d79029eb29913551 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-20 12:54:48,264 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=A 2023-04-20 12:54:48,264 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:48,264 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=B 2023-04-20 12:54:48,264 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:48,264 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=C 2023-04-20 12:54:48,264 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:48,308 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681995288287, max timestamp=1681995288307Num uniques 859; 2023-04-20 12:54:48,308 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 12:54:48,310 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681995288287, max timestamp=1681995288307Num uniques 859; 2023-04-20 12:54:48,310 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 12:54:48,310 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681995288287, max timestamp=1681995288307Num uniques 859; 2023-04-20 12:54:48,310 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 12:54:48,344 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:48,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40560 deadline: 1681995348340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,345 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:48,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:48,345 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:48,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40554 deadline: 1681995348341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:48,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:48,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:48,348 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:48,348 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:48,348 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:48,348 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:48,348 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,348 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:48,349 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:48,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:48,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:48,349 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:48,349 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,349 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:48,349 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:48,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:48,350 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:48,350 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:48,350 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:48,350 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:48,350 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:48,350 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:48,350 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:48,350 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:48,350 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:48,350 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:48,350 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:48,350 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:48,350 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:48,350 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:48,350 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,351 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:48,351 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:48,351 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,351 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:48,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:48,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:48,352 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,352 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:48,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:48,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:48,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:48,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:48,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c82db06[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:48,353 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:48,353 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,353 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,353 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:48,353 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,353 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:48,353 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:48,353 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:48,354 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,354 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:48,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:48,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:48,354 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:48,355 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:48,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:48,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:48,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:48,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:48,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:48,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:48,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:48,356 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:48,356 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:48,356 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:48,356 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:48,356 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:48,357 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,358 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:48,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:48,358 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:48,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:48,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:48,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:48,358 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:48,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:48,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:48,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,359 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:48,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:48,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,360 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,360 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:48,360 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:48,361 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:48,361 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:48,361 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:48,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:48,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:48,361 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,361 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,361 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,361 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:48,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:48,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:48,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:48,361 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:48,361 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:48,447 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:48,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40560 deadline: 1681995348445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,449 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:48,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40554 deadline: 1681995348446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,461 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:48,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:48,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:48,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:48,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:48,464 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:48,464 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:48,464 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:48,464 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:48,464 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,464 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:48,465 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,465 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:48,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:48,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:48,465 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,465 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:48,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:48,465 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:48,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:48,466 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:48,466 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:48,466 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:48,466 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:48,466 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:48,466 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:48,466 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:48,466 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:48,466 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:48,466 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:48,466 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:48,466 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:48,466 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:48,466 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:48,466 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,466 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:48,467 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:48,467 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,467 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:48,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:48,467 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,468 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:48,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:48,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:48,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:48,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:48,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:48,469 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:48,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36eff973[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:48,469 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,470 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,470 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,470 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:48,470 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:48,470 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:48,470 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:48,471 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:48,471 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:48,471 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:48,471 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:48,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:48,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:48,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:48,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:48,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:48,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:48,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:48,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:48,473 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:48,473 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:48,473 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:48,473 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:48,473 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:48,474 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:48,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:48,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:48,475 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:48,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:48,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:48,476 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:48,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:48,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,477 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,477 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:48,478 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:48,478 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,478 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,478 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,478 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,478 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:48,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,478 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:48,478 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:48,478 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:48,478 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:48,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:48,478 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:48,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:48,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:48,478 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:48,478 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:48,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:48,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:48,578 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:48,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:48,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:48,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:48,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:48,585 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:48,585 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:48,585 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:48,586 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:48,586 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,586 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:48,587 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:48,587 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:48,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:48,588 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,588 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:48,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:48,588 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:48,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:48,588 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:48,589 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:48,589 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:48,589 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:48,589 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:48,589 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:48,590 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:48,590 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:48,590 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:48,590 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:48,590 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:48,590 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:48,590 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:48,590 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:48,590 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,590 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:48,590 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:48,591 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,591 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:48,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:48,591 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,592 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:48,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:48,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:48,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:48,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:48,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:48,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75b1414b[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:48,593 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:48,593 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,594 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,594 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:48,594 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:48,594 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:48,594 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,594 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:48,596 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:48,596 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:48,596 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:48,596 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:48,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:48,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:48,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:48,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:48,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:48,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:48,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:48,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:48,599 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:48,599 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:48,599 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:48,599 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:48,599 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:48,601 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,602 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,603 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:48,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:48,603 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:48,604 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,604 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,604 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:48,604 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:48,604 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,604 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,604 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:48,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,604 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:48,604 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:48,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:48,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:48,604 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:48,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:48,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:48,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:48,604 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,605 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:48,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:48,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:48,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:48,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:48,606 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:48,651 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:48,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40560 deadline: 1681995348648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,651 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:48,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40554 deadline: 1681995348650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,671 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=420 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/832784cb368a493baf2129a852536900 2023-04-20 12:54:48,704 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:48,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:48,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:48,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:48,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:48,707 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:48,707 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:48,707 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:48,707 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:48,708 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,708 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:48,709 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:48,709 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:48,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:48,709 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,709 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:48,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:48,710 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:48,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:48,710 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:48,711 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:48,711 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:48,711 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:48,711 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:48,711 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:48,718 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:48,718 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:48,718 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:48,718 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:48,718 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:48,718 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:48,718 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:48,718 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:48,718 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,718 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:48,718 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:48,722 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,722 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:48,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:48,723 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,723 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:48,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:48,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:48,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:48,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:48,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:48,724 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:48,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34ac473a[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:48,724 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,732 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,732 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:48,732 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:48,732 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:48,732 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,732 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:48,739 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:48,739 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:48,739 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:48,739 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:48,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:48,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:48,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:48,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:48,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:48,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:48,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:48,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:48,741 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:48,741 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:48,741 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:48,741 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:48,741 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:48,748 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,748 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:48,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:48,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,749 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:48,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:48,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:48,749 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:48,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:48,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:48,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:48,750 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:48,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:48,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,756 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,756 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:48,756 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:48,756 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:48,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:48,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:48,756 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:48,756 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,756 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,756 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,756 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,756 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:48,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:48,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:48,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:48,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:48,757 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:48,757 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:48,857 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:48,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:48,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:48,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:48,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:48,868 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:48,868 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:48,868 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:48,868 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:48,868 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,868 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:48,869 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:48,869 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:48,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:48,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:48,870 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,870 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:48,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:48,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:48,870 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:48,878 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:48,878 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:48,878 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:48,878 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:48,878 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:48,880 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:48,880 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:48,880 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:48,880 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:48,880 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:48,880 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:48,880 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:48,880 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:48,880 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,880 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:48,880 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:48,882 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:48,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:48,882 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:48,883 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,883 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:48,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:48,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:48,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:48,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:48,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e5ce042[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:48,884 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:48,884 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,885 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,885 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:48,885 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:48,885 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:48,885 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,885 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:48,886 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:48,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:48,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:48,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:48,888 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:48,888 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:48,888 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:48,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:48,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:48,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:48,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:48,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:48,889 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:48,889 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:48,889 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:48,890 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:48,890 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:48,891 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,891 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:48,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:48,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:48,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,893 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:48,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:48,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:48,894 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:48,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:48,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:48,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,896 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:48,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:48,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,898 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:48,899 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:48,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:48,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:48,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:48,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:48,900 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,900 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:48,900 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:48,900 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,900 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,900 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:48,900 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:48,900 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:48,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:48,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:48,901 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:48,901 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:48,956 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:48,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40560 deadline: 1681995348952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:48,957 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:48,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40554 deadline: 1681995348953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,001 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:49,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:49,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:49,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:49,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:49,006 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:49,006 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:49,006 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:49,006 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:49,007 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:49,007 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:49,007 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:49,007 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:49,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:49,008 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,008 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:49,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:49,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:49,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:49,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:49,009 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:49,009 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:49,009 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:49,009 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:49,009 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:49,009 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:49,010 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:49,010 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:49,010 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:49,010 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:49,010 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:49,010 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:49,010 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:49,010 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:49,010 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,010 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:49,010 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:49,012 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,012 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:49,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:49,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:49,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:49,013 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:49,013 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:49,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:49,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:49,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:49,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:49,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5731a69b[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:49,015 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:49,015 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:49,015 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:49,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:49,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:49,016 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:49,016 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:49,016 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:49,016 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,016 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:49,017 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:49,017 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:49,017 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:49,017 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:49,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:49,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:49,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:49,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:49,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:49,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:49,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:49,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:49,020 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:49,020 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:49,020 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:49,021 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:49,021 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:49,022 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:49,022 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:49,022 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:49,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:49,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:49,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:49,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:49,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:49,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:49,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:49,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:49,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:49,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:49,023 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:49,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:49,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,025 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:49,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:49,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,026 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:49,026 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:49,026 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,026 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:49,026 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:49,026 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,026 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:49,026 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:49,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:49,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:49,027 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:49,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:49,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:49,027 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:49,026 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:49,027 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:49,027 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:49,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:49,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:49,087 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=420 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/8b6401c6284a41b6a399dee240007481 2023-04-20 12:54:49,127 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:49,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:49,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:49,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:49,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:49,130 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:49,130 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:49,130 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:49,130 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:49,130 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:49,130 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:49,132 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:49,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:49,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:49,132 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:49,133 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,133 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:49,133 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:49,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:49,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:49,134 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:49,134 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:49,134 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:49,134 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:49,134 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:49,134 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:49,134 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:49,134 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:49,134 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:49,134 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:49,134 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:49,134 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:49,135 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:49,135 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:49,135 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,135 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:49,135 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:49,136 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,136 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:49,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:49,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:49,136 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:49,136 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:49,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:49,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:49,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:49,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:49,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:49,137 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:49,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28f50e2f[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:49,137 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:49,138 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:49,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:49,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:49,138 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:49,138 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:49,138 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:49,139 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,139 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:49,139 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,139 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:49,139 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:49,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:49,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:49,139 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:49,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:49,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:49,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:49,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:49,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:49,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:49,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:49,141 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:49,141 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:49,141 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:49,141 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:49,141 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:49,142 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:49,142 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:49,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:49,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:49,142 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:49,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:49,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:49,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:49,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:49,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:49,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:49,142 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:49,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:49,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:49,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:49,143 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:49,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:49,145 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:49,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:49,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:49,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:49,145 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:49,145 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:49,145 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:49,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:49,145 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:49,145 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:49,145 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:49,145 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,145 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:49,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:49,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:49,145 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:49,146 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:49,146 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:49,246 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:49,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:49,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:49,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:49,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:49,249 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:49,250 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:49,250 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:49,250 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:49,250 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:49,250 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:49,251 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:49,251 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:49,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:49,251 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:49,251 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:49,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:49,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:49,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:49,252 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:49,252 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:49,252 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:49,252 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:49,252 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:49,252 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:49,252 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:49,252 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:49,252 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:49,252 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:49,253 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:49,253 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:49,253 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:49,253 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:49,253 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,253 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:49,253 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:49,254 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,254 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:49,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:49,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:49,254 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:49,254 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:49,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:49,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:49,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:49,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:49,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:49,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7eb6099c[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:49,255 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:49,256 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:49,256 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:49,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:49,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:49,256 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:49,256 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:49,256 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:49,256 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,256 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:49,257 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:49,257 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:49,257 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,257 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:49,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:49,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:49,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:49,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:49,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:49,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:49,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:49,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:49,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:49,259 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:49,259 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:49,259 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:49,259 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:49,259 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:49,260 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:49,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:49,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:49,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:49,260 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:49,260 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:49,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:49,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:49,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:49,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:49,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:49,261 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:49,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:49,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:49,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:49,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,262 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:49,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:49,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,264 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:49,264 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:49,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:49,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:49,264 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,264 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:49,264 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:49,264 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,264 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:49,264 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:49,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:49,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:49,264 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:49,264 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:49,264 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:49,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:49,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:49,264 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:49,265 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:49,365 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:49,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:49,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:49,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:49,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:49,368 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:49,368 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:49,368 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:49,368 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:49,369 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:49,369 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:49,370 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:49,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:49,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:49,371 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:49,371 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,371 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:49,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:49,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:49,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:49,371 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:49,372 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:49,372 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:49,372 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:49,372 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:49,372 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:49,374 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:49,374 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:49,374 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:49,374 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:49,374 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:49,374 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:49,374 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:49,374 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:49,374 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,374 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:49,374 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:49,390 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,390 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:49,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:49,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:49,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:49,391 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:49,391 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:49,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:49,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:49,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:49,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:49,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@711b8f8[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:49,393 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:49,393 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:49,397 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:49,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:49,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:49,398 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:49,398 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:49,398 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:49,398 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,398 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:49,399 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:49,399 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,399 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:49,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:49,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:49,399 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:49,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:49,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:49,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:49,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:49,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:49,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:49,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:49,402 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:49,402 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:49,402 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:49,402 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:49,402 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:49,403 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:49,404 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:49,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:49,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:49,404 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:49,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:49,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:49,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:49,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:49,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:49,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:49,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:49,405 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:49,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:49,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:49,407 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:49,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:49,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,408 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,408 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:49,408 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:49,408 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:49,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:49,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:49,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:49,409 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:49,409 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:49,408 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,409 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:49,409 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:49,409 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:49,409 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:49,409 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:49,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:49,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:49,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:49,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:49,462 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:49,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40554 deadline: 1681995349461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,463 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:49,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40560 deadline: 1681995349461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,505 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=420 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/79f282fd60674c50a588a5433f0a79ef 2023-04-20 12:54:49,509 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:49,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:49,510 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/832784cb368a493baf2129a852536900 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/832784cb368a493baf2129a852536900 2023-04-20 12:54:49,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:49,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:49,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:49,513 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:49,513 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:49,513 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:49,513 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:49,514 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:49,514 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:49,515 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:49,515 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:49,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:49,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:49,515 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,515 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:49,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:49,516 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/832784cb368a493baf2129a852536900, entries=250, sequenceid=420, filesize=16.7 K 2023-04-20 12:54:49,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:49,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:49,516 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:49,517 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:49,517 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:49,517 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:49,517 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:49,517 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:49,517 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:49,517 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:49,517 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/8b6401c6284a41b6a399dee240007481 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/8b6401c6284a41b6a399dee240007481 2023-04-20 12:54:49,517 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:49,517 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:49,517 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:49,517 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:49,517 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:49,517 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:49,517 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,517 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:49,517 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:49,519 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:49,519 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:49,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:49,519 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:49,519 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:49,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:49,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:49,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:49,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:49,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:49,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dcec3ff[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:49,521 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:49,521 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:49,521 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:49,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:49,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:49,522 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,522 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:49,522 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:49,522 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:49,522 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:49,523 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:49,523 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:49,523 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:49,523 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/8b6401c6284a41b6a399dee240007481, entries=150, sequenceid=420, filesize=11.9 K 2023-04-20 12:54:49,524 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:49,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:49,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:49,525 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/79f282fd60674c50a588a5433f0a79ef as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/79f282fd60674c50a588a5433f0a79ef 2023-04-20 12:54:49,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:49,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:49,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:49,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:49,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:49,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:49,527 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:49,527 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:49,527 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:49,527 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:49,527 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:49,531 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/79f282fd60674c50a588a5433f0a79ef, entries=150, sequenceid=420, filesize=11.9 K 2023-04-20 12:54:49,532 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 969c768384097479d79029eb29913551 in 1269ms, sequenceid=420, compaction requested=true 2023-04-20 12:54:49,532 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:49,532 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:49,532 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 12:54:49,532 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 12:54:49,532 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:54:49,533 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:54:49,534 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49358 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 12:54:49,534 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): 969c768384097479d79029eb29913551/B is initiating minor compaction (all files) 2023-04-20 12:54:49,535 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 969c768384097479d79029eb29913551/B in TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:49,534 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54238 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 12:54:49,535 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/8c04bf595cb54203a35d56d7b6585511, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/6752be952a5b4e309061525c71d0a106, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/577d6977dc2f47eea5af5a7d98974f5e, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/8b6401c6284a41b6a399dee240007481] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp, totalSize=48.2 K 2023-04-20 12:54:49,535 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): 969c768384097479d79029eb29913551/A is initiating minor compaction (all files) 2023-04-20 12:54:49,535 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 969c768384097479d79029eb29913551/A in TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:49,535 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/b1e672be0ab24c2e932b7d0e02e0133a, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/29d0ee9899d34e0099d4a7746913f829, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/27e82770a2fb4bdba87e4f2a551cb15c, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/832784cb368a493baf2129a852536900] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp, totalSize=53.0 K 2023-04-20 12:54:49,535 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 8c04bf595cb54203a35d56d7b6585511, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1681995286663 2023-04-20 12:54:49,535 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting b1e672be0ab24c2e932b7d0e02e0133a, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1681995286663 2023-04-20 12:54:49,535 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 6752be952a5b4e309061525c71d0a106, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=397, earliestPutTs=1681995286998 2023-04-20 12:54:49,536 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 29d0ee9899d34e0099d4a7746913f829, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=397, earliestPutTs=1681995286998 2023-04-20 12:54:49,536 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 577d6977dc2f47eea5af5a7d98974f5e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=409, earliestPutTs=1681995287641 2023-04-20 12:54:49,536 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:49,536 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:49,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:49,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:49,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:49,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:49,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:49,536 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:49,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:49,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:49,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:49,537 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 8b6401c6284a41b6a399dee240007481, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1681995288260 2023-04-20 12:54:49,537 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 27e82770a2fb4bdba87e4f2a551cb15c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=409, earliestPutTs=1681995287641 2023-04-20 12:54:49,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:49,537 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:49,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:49,538 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 832784cb368a493baf2129a852536900, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1681995288258 2023-04-20 12:54:49,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:49,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,538 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:49,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:49,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,548 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): 969c768384097479d79029eb29913551#B#compaction#247 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:49,549 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,549 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:49,549 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:49,549 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:49,549 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:49,549 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,549 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:49,549 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:49,549 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:49,549 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:49,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:49,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:49,550 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:49,550 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:49,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:49,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:49,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:49,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:49,551 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 969c768384097479d79029eb29913551#A#compaction#248 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:49,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:49,575 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/a9e39d2252ac43d68277aa7c7828ceca as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/a9e39d2252ac43d68277aa7c7828ceca 2023-04-20 12:54:49,580 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 969c768384097479d79029eb29913551/A of 969c768384097479d79029eb29913551 into a9e39d2252ac43d68277aa7c7828ceca(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:54:49,580 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:49,580 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551., storeName=969c768384097479d79029eb29913551/A, priority=12, startTime=1681995289532; duration=0sec 2023-04-20 12:54:49,580 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:54:49,580 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 12:54:49,581 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49358 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 12:54:49,581 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): 969c768384097479d79029eb29913551/C is initiating minor compaction (all files) 2023-04-20 12:54:49,581 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 969c768384097479d79029eb29913551/C in TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:49,581 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/bfae4ce02339452faa94eaa462bc031c, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/0a77acd8cdd1474489e5416eaf89fd5d, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/b731c58d0de249258e2a7dca947a08a1, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/79f282fd60674c50a588a5433f0a79ef] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp, totalSize=48.2 K 2023-04-20 12:54:49,581 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting bfae4ce02339452faa94eaa462bc031c, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1681995286663 2023-04-20 12:54:49,582 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 0a77acd8cdd1474489e5416eaf89fd5d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=397, earliestPutTs=1681995286998 2023-04-20 12:54:49,582 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting b731c58d0de249258e2a7dca947a08a1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=409, earliestPutTs=1681995287641 2023-04-20 12:54:49,582 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 79f282fd60674c50a588a5433f0a79ef, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1681995288260 2023-04-20 12:54:49,591 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 969c768384097479d79029eb29913551#C#compaction#249 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:49,650 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:49,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:49,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:49,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:49,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:49,654 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:49,654 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:49,654 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:49,654 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:49,655 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:49,655 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:49,656 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:49,656 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:49,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:49,657 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:49,657 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:49,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:49,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:49,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:49,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:49,657 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:49,662 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:49,662 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:49,662 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:49,662 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:49,662 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:49,670 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:49,670 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:49,670 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:49,670 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:49,670 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 969c768384097479d79029eb29913551 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-20 12:54:49,671 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=A 2023-04-20 12:54:49,671 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:54:49,671 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=B 2023-04-20 12:54:49,671 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:54:49,672 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=C 2023-04-20 12:54:49,672 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:54:49,687 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=456 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/3a1629a43bd94c74bee8893e8e6dff1e 2023-04-20 12:54:49,703 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=456 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/b85ff8ccd07f42688d57336fdcbbb58f 2023-04-20 12:54:49,959 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/b6a27d4d936f4a41a7d415a3ef4fdd6a as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/b6a27d4d936f4a41a7d415a3ef4fdd6a 2023-04-20 12:54:49,964 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 969c768384097479d79029eb29913551/B of 969c768384097479d79029eb29913551 into b6a27d4d936f4a41a7d415a3ef4fdd6a(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:54:49,964 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:49,964 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551., storeName=969c768384097479d79029eb29913551/B, priority=12, startTime=1681995289532; duration=0sec 2023-04-20 12:54:49,964 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:49,999 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/5e70ec1697cf41548a6ba4854633dccc as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/5e70ec1697cf41548a6ba4854633dccc 2023-04-20 12:54:50,004 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 969c768384097479d79029eb29913551/C of 969c768384097479d79029eb29913551 into 5e70ec1697cf41548a6ba4854633dccc(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:54:50,004 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:50,005 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551., storeName=969c768384097479d79029eb29913551/C, priority=12, startTime=1681995289533; duration=0sec 2023-04-20 12:54:50,005 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:50,122 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=456 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/63947960e67c49d1a4e0c6033c92dcdf 2023-04-20 12:54:50,132 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/3a1629a43bd94c74bee8893e8e6dff1e as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/3a1629a43bd94c74bee8893e8e6dff1e 2023-04-20 12:54:50,136 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/3a1629a43bd94c74bee8893e8e6dff1e, entries=150, sequenceid=456, filesize=11.9 K 2023-04-20 12:54:50,137 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/b85ff8ccd07f42688d57336fdcbbb58f as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/b85ff8ccd07f42688d57336fdcbbb58f 2023-04-20 12:54:50,141 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/b85ff8ccd07f42688d57336fdcbbb58f, entries=150, sequenceid=456, filesize=11.9 K 2023-04-20 12:54:50,141 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/63947960e67c49d1a4e0c6033c92dcdf as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/63947960e67c49d1a4e0c6033c92dcdf 2023-04-20 12:54:50,149 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/63947960e67c49d1a4e0c6033c92dcdf, entries=150, sequenceid=456, filesize=11.9 K 2023-04-20 12:54:50,151 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=0 B/0 for 969c768384097479d79029eb29913551 in 481ms, sequenceid=456, compaction requested=false 2023-04-20 12:54:50,151 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:50,151 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:50,151 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:50,151 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:50,151 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,151 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:50,151 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:50,152 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:50,152 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:50,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:50,153 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:50,153 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:50,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:50,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:50,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:50,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:50,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:50,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dca2a1f[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:50,155 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:50,155 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:50,156 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:50,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:50,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:50,156 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:50,156 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:50,156 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:50,157 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,157 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:50,157 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:50,157 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,157 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:50,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:50,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:50,157 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:50,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:50,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:50,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:50,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:50,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:50,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:50,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:50,160 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:50,160 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:50,160 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:50,160 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:50,160 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:50,161 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,161 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:50,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:50,161 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:50,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:50,162 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:50,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:50,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:50,164 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:50,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:50,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,178 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,178 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:50,178 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:50,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:50,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:50,178 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:50,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:50,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:50,178 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:50,178 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:50,178 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:50,178 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:50,178 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,178 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:50,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:50,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:50,178 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:50,178 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,278 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:50,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:50,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:50,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:50,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:50,282 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:50,282 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:50,282 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:50,283 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:50,283 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,283 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:50,284 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,284 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:50,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:50,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:50,284 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,284 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:50,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:50,284 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:50,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:50,285 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:50,285 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:50,285 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:50,285 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:50,285 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:50,285 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:50,285 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:50,285 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:50,286 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:50,286 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:50,286 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:50,286 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:50,286 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:50,286 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:50,286 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,286 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:50,286 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:50,287 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:50,287 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:50,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:50,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:50,287 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:50,288 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:50,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:50,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@137e32d3[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:50,302 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:50,302 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:50,303 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:50,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:50,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:50,303 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,303 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:50,303 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:50,303 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:50,303 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:50,304 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:50,304 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:50,304 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,304 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:50,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:50,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:50,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:50,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:50,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:50,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:50,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:50,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:50,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:50,308 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:50,308 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:50,308 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:50,308 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:50,308 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:50,310 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,310 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,310 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:50,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:50,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:50,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:50,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:50,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:50,311 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:50,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:50,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:50,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,312 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:50,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:50,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,314 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,314 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:50,314 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:50,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:50,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:50,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:50,314 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:50,315 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:50,314 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:50,314 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:50,315 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:50,314 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:50,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:50,315 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:50,315 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:50,315 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:50,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:50,415 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:50,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:50,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:50,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:50,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:50,418 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:50,418 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:50,418 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:50,418 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:50,419 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,419 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:50,420 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:50,420 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:50,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:50,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:50,421 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,421 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:50,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:50,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:50,421 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:50,434 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:50,434 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:50,434 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:50,434 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:50,434 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:50,434 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:50,434 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:50,435 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:50,435 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:50,443 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:50,443 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:50,443 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:50,443 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:50,444 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,444 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:50,444 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:50,445 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:50,445 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:50,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:50,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:50,445 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:50,446 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:50,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:50,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:50,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:50,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:50,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19229e3c[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:50,447 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:50,447 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:50,448 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:50,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:50,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:50,448 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:50,448 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:50,448 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:50,448 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,448 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:50,449 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,449 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:50,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:50,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:50,449 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:50,449 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:50,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:50,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:50,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:50,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:50,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:50,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:50,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:50,452 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:50,452 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:50,452 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:50,452 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:50,452 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:50,454 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,454 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,454 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:50,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:50,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:50,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:50,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:50,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:50,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:50,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:50,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:50,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,456 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:50,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:50,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,460 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,469 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,469 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:50,469 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:50,470 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,470 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:50,470 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:50,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:50,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:50,470 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:50,470 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,470 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:50,470 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:50,470 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:50,470 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:50,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:50,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:50,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:50,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:50,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 969c768384097479d79029eb29913551 2023-04-20 12:54:50,473 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 969c768384097479d79029eb29913551 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-20 12:54:50,473 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=A 2023-04-20 12:54:50,474 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:50,474 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=B 2023-04-20 12:54:50,474 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:50,474 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=C 2023-04-20 12:54:50,474 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:50,491 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=471 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/2d35bed43ed64ca0bf23cbeab53ce7f1 2023-04-20 12:54:50,509 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681995290491, max timestamp=1681995290507Num uniques 859; 2023-04-20 12:54:50,510 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 12:54:50,510 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681995290491, max timestamp=1681995290507Num uniques 859; 2023-04-20 12:54:50,510 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 12:54:50,511 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681995290491, max timestamp=1681995290507Num uniques 859; 2023-04-20 12:54:50,511 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 12:54:50,552 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:50,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40560 deadline: 1681995350549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,556 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:50,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40554 deadline: 1681995350553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,570 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:50,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:50,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:50,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:50,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:50,590 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:50,590 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:50,590 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:50,590 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:50,590 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,590 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:50,591 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,591 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:50,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:50,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:50,591 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,591 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:50,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:50,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:50,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:50,592 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:50,594 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:50,594 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:50,594 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:50,594 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:50,594 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:50,594 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:50,594 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:50,594 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:50,594 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:50,594 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:50,595 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:50,595 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:50,595 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:50,595 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,595 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:50,595 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:50,596 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,596 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:50,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:50,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:50,596 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:50,596 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:50,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:50,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:50,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:50,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:50,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:50,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@325867dc[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:50,598 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:50,598 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:50,598 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:50,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:50,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:50,598 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,598 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:50,598 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:50,598 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:50,598 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:50,599 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:50,599 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:50,599 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:50,599 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:50,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:50,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:50,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:50,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:50,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:50,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:50,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:50,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:50,604 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:50,604 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:50,604 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:50,604 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:50,604 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:50,607 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,607 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:50,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:50,607 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:50,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:50,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:50,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:50,608 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:50,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:50,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:50,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,609 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:50,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:50,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,615 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,616 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:50,616 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:50,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:50,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:50,616 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:50,616 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:50,616 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:50,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:50,616 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:50,616 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:50,616 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:50,616 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:50,616 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:50,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:50,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:50,616 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,655 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:50,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40560 deadline: 1681995350654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,659 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:50,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40554 deadline: 1681995350658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,716 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:50,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:50,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:50,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:50,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:50,719 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:50,719 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:50,719 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:50,719 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:50,720 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,720 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:50,720 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:50,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:50,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:50,720 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,721 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,721 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:50,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:50,721 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:50,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:50,721 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:50,721 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:50,721 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:50,722 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:50,722 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:50,722 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:50,722 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:50,722 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:50,722 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:50,722 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:50,722 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:50,722 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:50,722 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:50,722 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:50,722 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,722 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:50,722 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:50,723 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,723 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:50,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:50,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:50,730 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:50,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:50,730 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:50,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:50,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:50,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:50,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:50,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ece295a[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:50,732 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:50,732 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:50,736 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:50,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:50,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:50,736 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:50,736 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:50,736 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:50,737 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,737 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:50,737 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:50,737 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:50,737 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,737 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:50,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:50,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:50,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:50,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:50,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:50,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:50,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:50,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:50,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:50,740 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:50,740 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:50,740 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:50,740 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:50,740 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:50,741 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,741 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,741 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:50,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:50,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:50,741 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:50,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:50,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:50,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:50,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:50,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:50,742 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:50,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:50,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,744 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,744 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:50,744 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:50,744 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,744 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:50,744 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:50,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:50,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:50,745 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:50,745 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,745 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:50,745 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:50,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:50,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:50,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:50,745 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:50,745 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:50,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:50,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:50,845 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:50,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:50,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:50,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:50,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:50,848 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:50,848 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:50,848 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:50,848 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:50,849 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,849 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:50,850 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,850 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:50,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:50,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:50,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:50,851 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,851 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:50,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:50,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:50,852 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:50,859 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:50,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40560 deadline: 1681995350856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,862 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:50,862 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:50,862 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:50,862 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:50,862 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:50,862 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:50,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40554 deadline: 1681995350860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,876 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:50,876 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:50,881 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:50,881 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:50,881 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:50,881 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:50,881 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:50,881 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:50,881 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,881 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:50,881 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:50,883 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:50,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:50,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:50,883 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:50,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:50,883 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:50,884 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:50,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:50,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:50,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:50,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cc46cfa[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:50,885 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:50,885 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:50,886 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:50,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:50,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:50,886 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:50,886 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:50,886 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:50,887 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,887 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:50,887 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,887 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:50,887 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:50,887 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:50,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:50,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:50,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:50,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:50,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:50,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:50,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:50,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:50,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:50,890 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:50,890 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:50,890 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:50,890 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:50,891 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:50,893 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,893 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,893 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:50,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:50,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:50,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:50,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:50,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:50,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:50,894 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:50,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:50,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:50,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:50,905 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,907 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:50,907 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:50,907 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:50,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:50,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:50,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:50,907 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:50,908 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:50,908 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:50,908 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:50,908 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:50,907 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:50,908 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:50,908 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:50,908 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:50,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:50,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:50,940 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=471 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/a816b0a0a45648188ef6932137a5dbe3 2023-04-20 12:54:51,008 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:51,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:51,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:51,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:51,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:51,022 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:51,022 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:51,022 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:51,022 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:51,022 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,023 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:51,023 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:51,023 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:51,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:51,024 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,024 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:51,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:51,024 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:51,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:51,025 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:51,032 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:51,032 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:51,032 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:51,032 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:51,032 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:51,032 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:51,033 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:51,033 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:51,033 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:51,033 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:51,033 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:51,033 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:51,033 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:51,033 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,033 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:51,033 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:51,036 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:51,036 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:51,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:51,036 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:51,036 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:51,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:51,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:51,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:51,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:51,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:51,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@519ada20[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:51,038 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:51,038 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:51,039 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,039 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:51,040 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:51,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:51,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:51,040 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:51,040 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:51,040 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:51,041 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,041 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:51,041 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:51,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:51,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:51,041 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:51,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:51,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:51,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:51,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:51,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:51,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:51,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:51,044 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:51,044 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:51,044 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:51,044 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:51,044 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:51,045 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,045 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:51,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,045 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:51,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:51,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:51,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:51,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:51,046 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:51,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:51,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:51,047 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:51,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:51,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,049 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:51,049 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:51,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:51,049 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:51,049 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:51,049 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:51,049 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,049 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:51,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:51,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:51,050 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:51,050 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:51,050 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:51,049 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:51,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:51,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:51,050 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:51,050 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,150 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:51,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:51,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:51,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:51,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:51,156 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:51,156 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:51,156 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:51,156 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:51,157 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,157 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:51,157 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,157 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:51,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:51,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:51,158 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,158 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:51,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:51,158 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:51,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:51,159 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:51,160 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:51,160 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:51,160 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:51,160 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:51,160 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:51,161 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:51,161 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:51,161 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:51,161 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:51,161 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:51,161 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:51,161 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:51,161 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:51,161 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,161 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:51,161 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:51,162 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:51,162 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:51,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:51,162 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:51,162 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:51,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:51,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:51,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:51,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:51,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:51,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@200d3148[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:51,164 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:51,164 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:51,164 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:51,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:51,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:51,164 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:51,164 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:51,164 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:51,165 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,165 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:51,165 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:51,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40560 deadline: 1681995351162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,166 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:51,166 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:51,166 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:51,166 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:51,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:51,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:51,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:51,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:51,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:51,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:51,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:51,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:51,169 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:51,169 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:51,169 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:51,169 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:51,169 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:51,169 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:51,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40554 deadline: 1681995351166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,174 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,174 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,174 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:51,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:51,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:51,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,175 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:51,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:51,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:51,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:51,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:51,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:51,176 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:51,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:51,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,183 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,183 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:51,184 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:51,184 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:51,184 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:51,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:51,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:51,184 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:51,184 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,184 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:51,184 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:51,184 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,184 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:51,184 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:51,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:51,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:51,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:51,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:51,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:51,284 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:51,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:51,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:51,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:51,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:51,287 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:51,287 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:51,287 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:51,287 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:51,287 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,287 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:51,288 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:51,288 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:51,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:51,288 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,288 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:51,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:51,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:51,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:51,289 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:51,289 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:51,289 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:51,289 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:51,289 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:51,289 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:51,290 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:51,290 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:51,290 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:51,290 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:51,290 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:51,290 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:51,290 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:51,290 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:51,290 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,290 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:51,290 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:51,291 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,291 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:51,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:51,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:51,292 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:51,292 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:51,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:51,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:51,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:51,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:51,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:51,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fc658ea[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:51,295 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:51,296 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:51,311 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,311 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:51,313 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:51,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:51,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:51,314 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:51,314 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:51,314 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:51,316 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,316 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:51,316 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:51,316 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:51,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:51,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:51,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:51,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:51,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:51,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:51,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:51,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:51,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:51,319 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:51,319 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:51,319 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:51,319 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:51,319 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:51,320 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,320 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:51,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,320 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:51,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:51,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:51,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:51,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,321 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:51,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:51,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:51,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:51,322 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:51,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:51,324 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,324 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:51,324 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:51,324 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,324 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:51,324 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:51,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:51,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:51,324 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:51,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:51,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:51,324 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:51,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:51,324 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:51,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:51,324 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:51,324 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,324 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:51,325 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:51,351 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=471 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/0df0aae0a86046e6bdb42a9404784d2b 2023-04-20 12:54:51,355 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/2d35bed43ed64ca0bf23cbeab53ce7f1 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/2d35bed43ed64ca0bf23cbeab53ce7f1 2023-04-20 12:54:51,362 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/2d35bed43ed64ca0bf23cbeab53ce7f1, entries=200, sequenceid=471, filesize=14.3 K 2023-04-20 12:54:51,364 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/a816b0a0a45648188ef6932137a5dbe3 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/a816b0a0a45648188ef6932137a5dbe3 2023-04-20 12:54:51,372 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/a816b0a0a45648188ef6932137a5dbe3, entries=150, sequenceid=471, filesize=11.9 K 2023-04-20 12:54:51,373 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/0df0aae0a86046e6bdb42a9404784d2b as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/0df0aae0a86046e6bdb42a9404784d2b 2023-04-20 12:54:51,381 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/0df0aae0a86046e6bdb42a9404784d2b, entries=150, sequenceid=471, filesize=11.9 K 2023-04-20 12:54:51,382 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 969c768384097479d79029eb29913551 in 909ms, sequenceid=471, compaction requested=true 2023-04-20 12:54:51,382 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:51,382 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:51,382 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:54:51,382 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:54:51,382 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:54:51,383 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:54:51,384 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:54:51,384 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): 969c768384097479d79029eb29913551/B is initiating minor compaction (all files) 2023-04-20 12:54:51,384 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39748 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:54:51,384 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 969c768384097479d79029eb29913551/B in TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:51,384 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): 969c768384097479d79029eb29913551/A is initiating minor compaction (all files) 2023-04-20 12:54:51,384 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/b6a27d4d936f4a41a7d415a3ef4fdd6a, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/b85ff8ccd07f42688d57336fdcbbb58f, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/a816b0a0a45648188ef6932137a5dbe3] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp, totalSize=36.4 K 2023-04-20 12:54:51,384 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 969c768384097479d79029eb29913551/A in TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:51,384 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/a9e39d2252ac43d68277aa7c7828ceca, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/3a1629a43bd94c74bee8893e8e6dff1e, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/2d35bed43ed64ca0bf23cbeab53ce7f1] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp, totalSize=38.8 K 2023-04-20 12:54:51,384 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting b6a27d4d936f4a41a7d415a3ef4fdd6a, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1681995288260 2023-04-20 12:54:51,385 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting a9e39d2252ac43d68277aa7c7828ceca, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1681995288260 2023-04-20 12:54:51,385 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 3a1629a43bd94c74bee8893e8e6dff1e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1681995288337 2023-04-20 12:54:51,385 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting b85ff8ccd07f42688d57336fdcbbb58f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1681995288337 2023-04-20 12:54:51,389 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting a816b0a0a45648188ef6932137a5dbe3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=471, earliestPutTs=1681995290469 2023-04-20 12:54:51,390 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 2d35bed43ed64ca0bf23cbeab53ce7f1, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=471, earliestPutTs=1681995290469 2023-04-20 12:54:51,419 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): 969c768384097479d79029eb29913551#B#compaction#256 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:51,424 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 969c768384097479d79029eb29913551#A#compaction#257 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:51,425 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:51,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:51,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:51,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:51,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:51,446 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:51,446 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:51,446 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:51,446 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:51,448 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,448 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:51,449 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:51,449 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:51,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:51,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:51,450 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,450 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:51,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:51,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:51,450 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:51,451 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:51,451 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:51,451 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:51,451 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:51,451 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:51,451 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:51,451 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:51,451 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:51,451 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:51,452 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 969c768384097479d79029eb29913551 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-20 12:54:51,452 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=A 2023-04-20 12:54:51,453 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:54:51,453 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=B 2023-04-20 12:54:51,453 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:54:51,453 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=C 2023-04-20 12:54:51,453 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:54:51,471 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/d9fa90dbd864445c9e92049307912677 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/d9fa90dbd864445c9e92049307912677 2023-04-20 12:54:51,480 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 969c768384097479d79029eb29913551/B of 969c768384097479d79029eb29913551 into d9fa90dbd864445c9e92049307912677(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:54:51,480 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:51,480 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551., storeName=969c768384097479d79029eb29913551/B, priority=13, startTime=1681995291382; duration=0sec 2023-04-20 12:54:51,480 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:54:51,480 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:54:51,481 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:54:51,481 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): 969c768384097479d79029eb29913551/C is initiating minor compaction (all files) 2023-04-20 12:54:51,481 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 969c768384097479d79029eb29913551/C in TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:51,481 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/5e70ec1697cf41548a6ba4854633dccc, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/63947960e67c49d1a4e0c6033c92dcdf, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/0df0aae0a86046e6bdb42a9404784d2b] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp, totalSize=36.4 K 2023-04-20 12:54:51,482 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 5e70ec1697cf41548a6ba4854633dccc, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1681995288260 2023-04-20 12:54:51,482 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 63947960e67c49d1a4e0c6033c92dcdf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1681995288337 2023-04-20 12:54:51,482 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 0df0aae0a86046e6bdb42a9404784d2b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=471, earliestPutTs=1681995290469 2023-04-20 12:54:51,490 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=507 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/ad15889485ae4001967c0b1d401a496e 2023-04-20 12:54:51,491 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): 969c768384097479d79029eb29913551#C#compaction#259 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:51,493 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/9cb53b653ed54598b808171c0e93c6f5 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/9cb53b653ed54598b808171c0e93c6f5 2023-04-20 12:54:51,506 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 969c768384097479d79029eb29913551/A of 969c768384097479d79029eb29913551 into 9cb53b653ed54598b808171c0e93c6f5(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:54:51,506 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:51,506 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551., storeName=969c768384097479d79029eb29913551/A, priority=13, startTime=1681995291382; duration=0sec 2023-04-20 12:54:51,506 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:51,518 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/9b4e8e73339c4d34bd5a7d2d164f1f3b as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/9b4e8e73339c4d34bd5a7d2d164f1f3b 2023-04-20 12:54:51,526 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 969c768384097479d79029eb29913551/C of 969c768384097479d79029eb29913551 into 9b4e8e73339c4d34bd5a7d2d164f1f3b(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:54:51,527 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:51,527 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551., storeName=969c768384097479d79029eb29913551/C, priority=13, startTime=1681995291383; duration=0sec 2023-04-20 12:54:51,527 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:51,534 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=507 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/7022386c71304ee2be60687beba29aac 2023-04-20 12:54:51,584 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=507 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/5f7d56760e5644b387db282ec2de5143 2023-04-20 12:54:51,587 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/ad15889485ae4001967c0b1d401a496e as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/ad15889485ae4001967c0b1d401a496e 2023-04-20 12:54:51,590 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/ad15889485ae4001967c0b1d401a496e, entries=150, sequenceid=507, filesize=11.9 K 2023-04-20 12:54:51,591 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/7022386c71304ee2be60687beba29aac as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/7022386c71304ee2be60687beba29aac 2023-04-20 12:54:51,594 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/7022386c71304ee2be60687beba29aac, entries=150, sequenceid=507, filesize=11.9 K 2023-04-20 12:54:51,595 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/5f7d56760e5644b387db282ec2de5143 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/5f7d56760e5644b387db282ec2de5143 2023-04-20 12:54:51,598 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/5f7d56760e5644b387db282ec2de5143, entries=150, sequenceid=507, filesize=11.9 K 2023-04-20 12:54:51,599 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=0 B/0 for 969c768384097479d79029eb29913551 in 147ms, sequenceid=507, compaction requested=false 2023-04-20 12:54:51,599 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:51,599 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:51,599 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:51,599 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:51,599 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,600 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:51,600 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:51,601 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,601 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:51,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:51,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:51,601 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:51,601 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:51,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:51,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:51,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:51,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:51,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:51,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@109a5a54[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:51,604 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:51,604 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:51,605 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:51,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:51,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:51,606 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:51,606 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:51,606 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:51,606 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,606 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:51,607 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,607 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:51,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:51,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:51,607 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:51,607 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:51,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:51,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:51,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:51,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:51,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:51,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:51,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:51,611 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:51,611 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:51,611 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:51,611 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:51,611 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:51,619 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,619 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:51,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:51,619 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:51,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:51,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:51,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:51,620 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:51,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:51,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:51,622 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:51,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:51,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,632 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,632 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:51,632 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:51,632 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,632 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:51,632 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:51,632 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:51,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:51,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:51,632 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:51,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:51,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:51,632 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:51,633 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:51,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:51,633 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:51,633 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:51,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 969c768384097479d79029eb29913551 2023-04-20 12:54:51,680 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 969c768384097479d79029eb29913551 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-20 12:54:51,680 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=A 2023-04-20 12:54:51,680 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:51,680 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=B 2023-04-20 12:54:51,680 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:51,680 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=C 2023-04-20 12:54:51,680 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:51,688 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=521 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/da06d81b86eb44a4b4afca9fa862bc34 2023-04-20 12:54:51,721 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:51,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681995291695, max timestamp=1681995291719Num uniques 859; 2023-04-20 12:54:51,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681995291695, max timestamp=1681995291719Num uniques 859; 2023-04-20 12:54:51,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 12:54:51,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40560 deadline: 1681995351720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681995291695, max timestamp=1681995291719Num uniques 859; 2023-04-20 12:54:51,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 12:54:51,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 12:54:51,733 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:51,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:51,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:51,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:51,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:51,736 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:51,736 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:51,736 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:51,736 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:51,737 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,737 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:51,737 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:51,738 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:51,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:51,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:51,738 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,738 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:51,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:51,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:51,739 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:51,740 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:51,740 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:51,740 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:51,741 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:51,741 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:51,742 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:51,742 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:51,742 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:51,742 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:51,742 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:51,742 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:51,742 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:51,742 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:51,742 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,742 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:51,742 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:51,743 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:51,744 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:51,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:51,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:51,746 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:51,746 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:51,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:51,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:51,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:51,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:51,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1756c9bd[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:51,755 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:51,755 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:51,758 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:51,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:51,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:51,758 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:51,758 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,758 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:51,758 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:51,758 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:51,759 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:51,759 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,759 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:51,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:51,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:51,759 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:51,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:51,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:51,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:51,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:51,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:51,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:51,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:51,763 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:51,763 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:51,763 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:51,763 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:51,763 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:51,764 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,766 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,768 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:51,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:51,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:51,768 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,768 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:51,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,768 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,768 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,768 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:51,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,768 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:51,768 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:51,768 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:51,768 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:51,768 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,768 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:51,768 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:51,768 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:51,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:51,768 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:51,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:51,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:51,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:51,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:51,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:51,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:51,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:51,770 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:51,777 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:51,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40554 deadline: 1681995351775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:51,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40560 deadline: 1681995351822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,868 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:51,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:51,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:51,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:51,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:51,872 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:51,872 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:51,872 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:51,872 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:51,872 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,872 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:51,874 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:51,874 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:51,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:51,874 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,874 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:51,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:51,875 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:51,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:51,875 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:51,875 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:51,875 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:51,876 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:51,876 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:51,876 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:51,876 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:51,877 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:51,877 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:51,877 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:51,877 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:51,877 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:51,877 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:51,877 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:51,877 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,877 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:51,877 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:51,878 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:51,878 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:51,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:51,878 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:51,878 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:51,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:51,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:51,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:51,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:51,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:51,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e182d6e[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:51,879 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:51,880 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:51,880 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:51,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40554 deadline: 1681995351880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,882 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:51,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:51,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:51,883 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:51,883 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:51,883 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:51,883 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,883 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:51,884 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,884 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:51,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:51,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:51,884 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:51,884 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:51,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:51,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:51,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:51,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:51,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:51,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:51,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:51,886 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:51,886 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:51,886 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:51,886 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:51,886 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:51,887 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,887 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:51,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:51,887 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:51,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:51,888 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:51,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:51,888 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:51,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:51,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:51,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,890 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,890 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:51,890 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:51,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:51,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:51,890 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:51,890 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:51,890 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,890 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:51,890 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:51,890 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,890 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:51,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:51,890 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:51,890 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:51,891 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:51,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:51,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:51,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:51,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:51,991 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:51,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:51,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:51,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:51,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:51,993 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:51,993 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:51,994 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:51,994 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:51,994 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,994 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:51,995 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:51,995 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:51,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:51,995 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,995 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:51,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:51,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:51,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:51,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:51,995 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:51,996 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:51,996 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:51,996 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:51,996 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:51,996 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:51,996 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:51,996 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:51,996 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:51,996 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:51,996 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:51,996 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:51,996 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:51,996 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:51,996 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,996 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:51,996 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:51,997 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,997 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:51,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:51,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:51,997 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:51,997 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:51,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:51,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:51,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:51,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:51,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:51,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:51,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c42f4f[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:51,999 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:51,999 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:52,000 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:52,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:52,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:52,000 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:52,000 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:52,000 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,000 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:52,000 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:52,001 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:52,001 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,001 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:52,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:52,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:52,001 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:52,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:52,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:52,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:52,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:52,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:52,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:52,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:52,003 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:52,003 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:52,003 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:52,003 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:52,003 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:52,004 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:52,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:52,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:52,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:52,004 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:52,004 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:52,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:52,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:52,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:52,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:52,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:52,005 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:52,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:52,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:52,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:52,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,006 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:52,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:52,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,008 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,008 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:52,008 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:52,008 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:52,008 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:52,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:52,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:52,008 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:52,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:52,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:52,009 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,009 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:52,009 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:52,009 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:52,009 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:52,009 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:52,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:52,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:52,027 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:52,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40560 deadline: 1681995352024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,084 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:52,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40554 deadline: 1681995352081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,109 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:52,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:52,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:52,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:52,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:52,112 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:52,112 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:52,112 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:52,112 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:52,112 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=521 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/2cc6856eeece4a1081bf9bca5a103c6c 2023-04-20 12:54:52,122 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:52,122 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:52,123 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:52,123 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:52,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:52,123 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,123 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:52,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:52,123 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:52,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:52,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:52,124 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:52,124 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:52,124 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:52,124 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:52,124 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:52,124 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:52,124 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:52,124 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:52,124 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:52,124 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:52,125 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:52,125 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:52,125 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:52,125 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:52,125 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,125 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:52,125 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:52,126 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,126 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:52,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:52,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:52,126 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:52,126 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:52,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:52,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:52,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:52,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:52,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:52,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@414e8d99[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:52,127 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:52,127 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:52,128 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:52,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:52,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:52,128 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:52,128 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,128 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:52,128 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:52,128 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:52,129 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:52,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:52,129 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:52,129 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:52,130 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:52,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:52,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:52,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:52,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:52,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:52,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:52,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:52,132 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:52,132 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:52,132 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:52,132 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:52,133 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:52,134 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:52,134 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:52,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:52,134 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:52,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:52,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:52,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:52,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:52,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:52,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:52,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:52,134 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:52,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:52,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:52,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:52,135 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:52,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:52,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,136 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,136 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:52,136 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:52,136 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:52,137 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:52,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:52,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:52,136 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,137 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:52,137 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:52,137 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:52,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:52,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:52,137 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:52,137 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:52,137 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:52,137 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:52,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:52,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:52,237 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:52,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:52,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:52,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:52,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:52,243 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:52,243 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:52,243 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:52,243 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:52,243 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:52,243 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:52,244 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:52,244 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:52,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:52,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:52,245 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,245 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:52,245 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:52,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:52,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:52,245 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:52,258 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:52,258 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:52,258 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:52,258 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:52,258 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:52,273 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:52,273 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:52,273 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:52,273 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:52,273 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:52,273 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:52,273 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:52,273 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:52,273 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,273 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:52,273 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:52,274 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:52,274 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:52,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:52,275 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:52,275 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:52,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:52,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:52,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:52,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:52,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:52,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6074bf2c[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:52,276 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:52,276 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:52,279 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:52,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:52,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:52,279 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:52,279 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,279 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:52,279 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:52,279 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:52,280 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,280 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:52,280 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:52,280 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:52,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:52,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:52,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:52,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:52,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:52,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:52,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:52,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:52,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:52,283 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:52,283 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:52,283 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:52,283 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:52,283 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:52,284 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:52,284 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:52,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:52,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:52,284 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:52,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:52,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:52,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:52,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:52,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:52,285 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:52,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:52,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:52,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:52,287 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:52,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:52,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:52,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,289 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:52,289 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:52,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:52,289 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:52,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:52,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:52,289 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:52,290 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:52,289 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:52,290 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:52,290 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:52,290 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,290 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:52,290 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:52,290 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:52,290 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:52,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:52,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:52,331 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:52,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40560 deadline: 1681995352330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,388 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:52,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40554 deadline: 1681995352387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,390 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:52,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:52,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:52,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:52,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:52,392 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:52,392 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:52,393 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:52,393 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:52,393 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:52,393 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:52,394 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:52,394 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:52,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:52,394 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,394 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:52,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:52,394 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:52,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:52,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:52,395 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:52,395 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:52,395 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:52,395 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:52,395 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:52,395 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:52,395 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:52,395 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:52,395 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:52,395 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:52,395 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:52,395 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:52,396 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:52,396 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:52,396 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,396 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:52,396 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:52,396 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:52,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:52,397 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:52,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:52,397 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:52,397 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:52,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:52,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:52,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:52,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:52,398 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:52,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f989d11[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:52,398 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:52,399 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:52,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:52,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:52,399 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:52,399 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:52,399 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:52,399 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,399 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:52,400 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,400 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:52,400 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:52,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:52,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:52,400 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:52,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:52,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:52,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:52,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:52,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:52,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:52,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:52,403 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:52,403 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:52,403 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:52,403 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:52,403 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:52,404 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:52,404 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:52,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:52,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:52,404 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:52,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:52,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:52,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:52,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:52,404 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:52,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:52,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:52,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:52,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:52,406 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:52,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:52,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:52,407 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:52,407 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:52,407 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,408 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:52,408 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:52,408 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,408 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:52,408 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:52,408 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:52,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:52,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:52,408 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:52,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:52,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:52,408 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:52,408 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:52,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:52,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:52,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:52,508 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:52,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:52,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:52,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:52,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:52,512 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:52,512 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:52,512 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:52,512 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:52,513 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:52,513 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:52,532 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=521 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/12181e248643468a92f079c4fc66498c 2023-04-20 12:54:52,538 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/da06d81b86eb44a4b4afca9fa862bc34 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/da06d81b86eb44a4b4afca9fa862bc34 2023-04-20 12:54:52,544 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/da06d81b86eb44a4b4afca9fa862bc34, entries=150, sequenceid=521, filesize=11.9 K 2023-04-20 12:54:52,545 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/2cc6856eeece4a1081bf9bca5a103c6c as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/2cc6856eeece4a1081bf9bca5a103c6c 2023-04-20 12:54:52,548 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/2cc6856eeece4a1081bf9bca5a103c6c, entries=150, sequenceid=521, filesize=11.9 K 2023-04-20 12:54:52,549 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/12181e248643468a92f079c4fc66498c as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/12181e248643468a92f079c4fc66498c 2023-04-20 12:54:52,552 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/12181e248643468a92f079c4fc66498c, entries=150, sequenceid=521, filesize=11.9 K 2023-04-20 12:54:52,553 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 969c768384097479d79029eb29913551 in 874ms, sequenceid=521, compaction requested=true 2023-04-20 12:54:52,553 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:52,553 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:52,553 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:54:52,553 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:54:52,553 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:52,553 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:54:52,554 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:54:52,554 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): 969c768384097479d79029eb29913551/A is initiating minor compaction (all files) 2023-04-20 12:54:52,554 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 969c768384097479d79029eb29913551/A in TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:52,555 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/9cb53b653ed54598b808171c0e93c6f5, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/ad15889485ae4001967c0b1d401a496e, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/da06d81b86eb44a4b4afca9fa862bc34] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp, totalSize=36.5 K 2023-04-20 12:54:52,555 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:54:52,555 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): 969c768384097479d79029eb29913551/B is initiating minor compaction (all files) 2023-04-20 12:54:52,555 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 969c768384097479d79029eb29913551/B in TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:52,555 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/d9fa90dbd864445c9e92049307912677, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/7022386c71304ee2be60687beba29aac, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/2cc6856eeece4a1081bf9bca5a103c6c] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp, totalSize=36.5 K 2023-04-20 12:54:52,555 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting d9fa90dbd864445c9e92049307912677, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=471, earliestPutTs=1681995290469 2023-04-20 12:54:52,555 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 9cb53b653ed54598b808171c0e93c6f5, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=471, earliestPutTs=1681995290469 2023-04-20 12:54:52,556 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 7022386c71304ee2be60687beba29aac, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=507, earliestPutTs=1681995290544 2023-04-20 12:54:52,556 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting ad15889485ae4001967c0b1d401a496e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=507, earliestPutTs=1681995290544 2023-04-20 12:54:52,556 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 2cc6856eeece4a1081bf9bca5a103c6c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=521, earliestPutTs=1681995291674 2023-04-20 12:54:52,556 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting da06d81b86eb44a4b4afca9fa862bc34, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=521, earliestPutTs=1681995291674 2023-04-20 12:54:52,565 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): 969c768384097479d79029eb29913551#B#compaction#265 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:52,565 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 969c768384097479d79029eb29913551#A#compaction#266 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:52,576 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:52,576 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:52,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:52,576 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:52,576 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:52,577 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:52,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:52,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:52,577 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:52,582 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:52,582 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:52,582 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:52,582 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:52,582 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:52,594 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:52,594 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:52,594 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:52,594 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:52,595 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 969c768384097479d79029eb29913551 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-20 12:54:52,595 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=A 2023-04-20 12:54:52,596 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:54:52,596 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=B 2023-04-20 12:54:52,596 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:54:52,596 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=C 2023-04-20 12:54:52,596 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:54:52,599 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/41f25a07113845e7946debf0069fccd1 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/41f25a07113845e7946debf0069fccd1 2023-04-20 12:54:52,603 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 969c768384097479d79029eb29913551/B of 969c768384097479d79029eb29913551 into 41f25a07113845e7946debf0069fccd1(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:54:52,603 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:52,604 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551., storeName=969c768384097479d79029eb29913551/B, priority=13, startTime=1681995292553; duration=0sec 2023-04-20 12:54:52,604 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:54:52,604 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:54:52,605 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:54:52,605 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): 969c768384097479d79029eb29913551/C is initiating minor compaction (all files) 2023-04-20 12:54:52,605 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 969c768384097479d79029eb29913551/C in TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:52,605 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/9b4e8e73339c4d34bd5a7d2d164f1f3b, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/5f7d56760e5644b387db282ec2de5143, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/12181e248643468a92f079c4fc66498c] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp, totalSize=36.5 K 2023-04-20 12:54:52,605 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 9b4e8e73339c4d34bd5a7d2d164f1f3b, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=471, earliestPutTs=1681995290469 2023-04-20 12:54:52,606 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 5f7d56760e5644b387db282ec2de5143, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=507, earliestPutTs=1681995290544 2023-04-20 12:54:52,606 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 12181e248643468a92f079c4fc66498c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=521, earliestPutTs=1681995291674 2023-04-20 12:54:52,611 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): 969c768384097479d79029eb29913551#C#compaction#268 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:52,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 969c768384097479d79029eb29913551 2023-04-20 12:54:52,839 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:52,860 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:52,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40560 deadline: 1681995352859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,893 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:52,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40554 deadline: 1681995352892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,963 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:52,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40560 deadline: 1681995352961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:52,995 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/0673a82ad94449fb8c9e6f990cedea6b as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/0673a82ad94449fb8c9e6f990cedea6b 2023-04-20 12:54:52,999 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 969c768384097479d79029eb29913551/A of 969c768384097479d79029eb29913551 into 0673a82ad94449fb8c9e6f990cedea6b(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:54:52,999 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:52,999 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551., storeName=969c768384097479d79029eb29913551/A, priority=13, startTime=1681995292553; duration=0sec 2023-04-20 12:54:52,999 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:53,008 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=557 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/234cc413c8c24ee193712af21a20a2fa 2023-04-20 12:54:53,020 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=557 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/953a4760cc0046c8acd0ee37e1f290e5 2023-04-20 12:54:53,026 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/a420974e5e064667bd2f9c26387afe80 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/a420974e5e064667bd2f9c26387afe80 2023-04-20 12:54:53,032 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 969c768384097479d79029eb29913551/C of 969c768384097479d79029eb29913551 into a420974e5e064667bd2f9c26387afe80(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:54:53,032 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:53,032 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551., storeName=969c768384097479d79029eb29913551/C, priority=13, startTime=1681995292553; duration=0sec 2023-04-20 12:54:53,032 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:53,168 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:53,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40560 deadline: 1681995353165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,439 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=557 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/b66b4e46a6fe47709a8791c32623d95a 2023-04-20 12:54:53,443 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/234cc413c8c24ee193712af21a20a2fa as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/234cc413c8c24ee193712af21a20a2fa 2023-04-20 12:54:53,446 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/234cc413c8c24ee193712af21a20a2fa, entries=150, sequenceid=557, filesize=11.9 K 2023-04-20 12:54:53,447 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/953a4760cc0046c8acd0ee37e1f290e5 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/953a4760cc0046c8acd0ee37e1f290e5 2023-04-20 12:54:53,451 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/953a4760cc0046c8acd0ee37e1f290e5, entries=150, sequenceid=557, filesize=11.9 K 2023-04-20 12:54:53,451 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/b66b4e46a6fe47709a8791c32623d95a as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/b66b4e46a6fe47709a8791c32623d95a 2023-04-20 12:54:53,464 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/b66b4e46a6fe47709a8791c32623d95a, entries=150, sequenceid=557, filesize=11.9 K 2023-04-20 12:54:53,465 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 969c768384097479d79029eb29913551 in 871ms, sequenceid=557, compaction requested=false 2023-04-20 12:54:53,466 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:53,466 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:53,466 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:53,466 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:53,466 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,466 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:53,466 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:53,467 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,467 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:53,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:53,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:53,467 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:53,467 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:53,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:53,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:53,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:53,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:53,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:53,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d846bfa[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:53,469 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:53,469 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:53,470 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:53,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:53,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:53,470 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:53,470 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:53,470 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:53,471 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,471 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:53,472 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:53,472 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:53,472 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:53,472 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:53,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:53,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:53,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:53,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:53,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 969c768384097479d79029eb29913551 2023-04-20 12:54:53,473 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 969c768384097479d79029eb29913551 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-20 12:54:53,473 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=A 2023-04-20 12:54:53,473 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:53,473 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=B 2023-04-20 12:54:53,474 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:53,474 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=C 2023-04-20 12:54:53,474 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:53,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:53,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:53,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:53,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:53,476 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:53,476 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:53,476 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:53,477 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:53,477 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:53,486 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:53,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:53,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:53,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:53,486 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:53,486 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:53,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:53,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:53,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:53,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:53,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:53,487 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:53,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:53,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:53,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:53,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,488 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:53,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:53,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,491 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:53,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:53,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:53,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:53,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:53,491 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:53,491 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:53,491 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:53,491 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:53,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:53,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:53,493 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,493 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:53,493 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:53,493 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,493 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:53,493 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:53,494 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:53,558 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:53,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40560 deadline: 1681995353554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,591 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:53,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:53,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:53,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:53,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:53,594 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:53,594 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:53,594 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:53,594 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:53,594 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:53,594 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:53,595 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,595 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,595 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:53,602 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:53,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:53,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:53,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:53,603 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:53,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:53,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:53,603 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:53,606 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:53,607 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:53,607 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:53,607 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:53,607 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:53,607 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:53,607 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:53,607 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:53,607 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:53,607 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:53,607 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:53,607 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:53,607 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:53,607 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,607 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:53,607 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:53,611 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,611 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:53,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:53,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:53,611 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:53,612 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:53,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:53,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:53,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:53,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:53,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:53,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68930796[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:53,613 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:53,613 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:53,614 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:53,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:53,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:53,614 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:53,614 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:53,614 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:53,614 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,614 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:53,616 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,616 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:53,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:53,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:53,616 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:53,616 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:53,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:53,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:53,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:53,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:53,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:53,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:53,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:53,618 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:53,618 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:53,618 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:53,618 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:53,619 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:53,620 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:53,620 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:53,620 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:53,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:53,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:53,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:53,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:53,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:53,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:53,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:53,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:53,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:53,621 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:53,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:53,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:53,623 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:53,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:53,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,625 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:53,625 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,625 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:53,625 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:53,626 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:53,626 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:53,626 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:53,626 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:53,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:53,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:53,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:53,626 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:53,626 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:53,626 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:53,630 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:53,630 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:53,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:53,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:53,662 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:53,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40560 deadline: 1681995353659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,730 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:53,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:53,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:53,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:53,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:53,742 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:53,742 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:53,742 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:53,742 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:53,742 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:53,742 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:53,743 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:53,743 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:53,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:53,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:53,744 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,744 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:53,744 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:53,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:53,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:53,744 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:53,750 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:53,750 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:53,750 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:53,750 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:53,750 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:53,750 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:53,750 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:53,750 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:53,750 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:53,750 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:53,750 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:53,751 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:53,751 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:53,751 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,751 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:53,751 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:53,757 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:53,757 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:53,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:53,758 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:53,758 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:53,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:53,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:53,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:53,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:53,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:53,759 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:53,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ddb5877[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:53,759 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:53,766 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:53,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:53,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:53,766 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,766 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:53,766 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:53,766 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:53,766 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:53,767 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,767 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:53,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:53,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:53,767 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:53,767 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:53,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:53,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:53,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:53,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:53,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:53,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:53,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:53,769 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:53,769 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:53,770 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:53,770 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:53,770 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:53,770 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:53,770 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:53,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:53,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:53,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:53,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:53,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:53,771 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:53,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:53,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:53,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:53,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:53,771 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:53,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:53,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:53,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,773 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:53,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:53,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,775 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,775 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:53,775 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:53,775 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,775 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:53,775 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:53,775 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:53,775 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:53,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:53,775 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:53,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:53,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:53,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:53,775 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:53,775 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:53,775 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:53,775 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:53,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:53,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:53,864 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:53,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40560 deadline: 1681995353863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,876 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:53,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:53,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:53,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:53,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:53,878 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:53,878 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:53,878 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:53,878 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:53,879 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:53,879 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:53,879 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:53,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:53,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:53,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:53,880 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:53,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:53,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:53,881 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:53,882 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:53,882 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:53,882 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:53,882 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:53,882 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:53,886 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,887 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,887 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:53,898 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:53,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40554 deadline: 1681995353896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,906 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:53,906 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:53,906 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:53,906 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:53,906 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:53,906 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:53,906 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:53,906 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:53,906 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,906 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:53,906 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:53,907 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,907 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:53,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:53,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:53,908 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:53,908 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:53,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:53,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:53,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:53,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:53,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:53,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@169028c8[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:53,909 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:53,910 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:53,910 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:53,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:53,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:53,910 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:53,910 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:53,910 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:53,911 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,911 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:53,911 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,911 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:53,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:53,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:53,911 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:53,911 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:53,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:53,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:53,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:53,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:53,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:53,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:53,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:53,913 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:53,913 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:53,913 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:53,913 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:53,913 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:53,915 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:53,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:53,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:53,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:53,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:53,915 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:53,915 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:53,915 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:53,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:53,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:53,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:53,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:53,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:53,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:53,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:53,917 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:53,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:53,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,928 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:53,928 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:53,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:53,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:53,928 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,928 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:53,928 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:53,928 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:53,928 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:53,928 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:53,928 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:53,928 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:53,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:53,929 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:53,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:53,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:53,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:53,929 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:53,929 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:53,936 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=572 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/42e016c4a8914e9b9b45f35c5d793427 2023-04-20 12:54:54,029 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:54,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:54,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:54,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:54,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:54,032 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:54,032 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:54,032 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:54,032 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:54,033 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,033 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:54,033 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:54,033 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:54,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:54,034 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,034 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:54,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:54,034 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:54,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:54,034 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:54,035 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:54,035 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:54,035 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:54,035 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:54,035 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:54,035 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:54,035 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:54,035 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:54,035 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:54,035 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:54,035 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:54,035 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:54,035 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:54,035 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,035 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:54,035 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:54,036 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,036 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:54,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:54,036 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:54,036 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:54,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:54,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:54,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:54,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:54,037 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:54,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68355a5b[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:54,037 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,038 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,038 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:54,038 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:54,038 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:54,038 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,038 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:54,039 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:54,039 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:54,039 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,039 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:54,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:54,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:54,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:54,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:54,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:54,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:54,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:54,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:54,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:54,041 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:54,041 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:54,041 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:54,041 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:54,041 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:54,041 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,041 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:54,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:54,041 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:54,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:54,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:54,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:54,042 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:54,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:54,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:54,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,043 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:54,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:54,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,045 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,045 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:54,045 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:54,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:54,045 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:54,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:54,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:54,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:54,045 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:54,046 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,046 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,045 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:54,046 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:54,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:54,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:54,046 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,046 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:54,046 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,146 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:54,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:54,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:54,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:54,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:54,149 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:54,149 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:54,149 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:54,149 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:54,149 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,149 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:54,150 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:54,150 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:54,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:54,150 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,150 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:54,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:54,151 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:54,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:54,151 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:54,151 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:54,151 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:54,151 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:54,151 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:54,151 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:54,151 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:54,151 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:54,151 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:54,151 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:54,152 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:54,152 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:54,152 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:54,152 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:54,152 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,152 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:54,152 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:54,152 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,152 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:54,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:54,153 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,153 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:54,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:54,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:54,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:54,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:54,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:54,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@186a5f7c[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:54,154 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:54,154 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,155 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,155 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,155 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:54,155 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:54,155 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:54,155 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:54,156 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:54,156 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,156 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:54,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:54,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:54,156 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:54,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:54,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:54,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:54,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:54,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:54,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:54,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:54,158 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:54,158 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:54,158 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:54,158 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:54,158 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:54,159 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,159 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,159 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:54,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:54,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:54,159 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:54,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:54,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:54,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:54,160 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:54,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:54,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:54,161 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:54,161 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:54,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:54,161 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:54,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:54,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:54,161 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:54,161 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:54,162 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:54,162 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,162 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:54,162 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:54,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:54,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:54,162 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,162 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,162 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:54,168 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:54,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40560 deadline: 1681995354166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,262 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:54,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:54,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:54,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:54,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:54,274 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:54,274 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:54,274 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:54,274 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:54,275 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,275 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:54,276 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:54,276 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:54,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:54,276 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,276 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:54,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:54,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:54,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:54,277 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:54,277 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:54,277 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:54,277 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:54,277 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:54,277 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:54,277 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:54,277 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:54,277 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:54,277 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:54,277 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:54,277 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:54,277 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:54,277 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:54,277 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,277 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:54,277 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:54,278 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,278 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:54,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:54,278 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,278 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:54,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:54,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:54,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:54,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:54,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:54,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f05a4e6[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:54,280 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:54,280 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,281 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,281 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:54,281 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:54,281 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:54,282 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,282 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:54,282 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:54,282 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:54,282 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:54,282 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:54,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:54,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:54,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:54,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:54,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:54,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:54,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:54,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:54,285 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:54,285 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:54,285 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:54,285 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:54,285 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:54,287 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,287 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:54,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:54,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,287 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:54,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,287 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:54,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:54,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:54,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:54,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:54,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:54,289 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:54,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:54,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,291 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:54,291 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:54,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:54,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:54,291 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:54,291 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:54,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:54,291 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:54,291 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:54,291 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,291 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,291 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,291 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,291 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:54,292 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:54,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:54,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:54,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:54,361 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=572 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/12edb73e88ab4c92aec1de9a76a5a56a 2023-04-20 12:54:54,392 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:54,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:54,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:54,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:54,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:54,398 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:54,398 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:54,398 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:54,398 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:54,398 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,399 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:54,399 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:54,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:54,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:54,400 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:54,400 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,400 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:54,401 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:54,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:54,401 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:54,401 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:54,401 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:54,401 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:54,401 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:54,401 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:54,402 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:54,402 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:54,402 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:54,402 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:54,402 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:54,402 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:54,402 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:54,403 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:54,403 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,403 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:54,403 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:54,403 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,404 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:54,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:54,404 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,404 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:54,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:54,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:54,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:54,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:54,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:54,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78c417a6[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:54,406 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:54,406 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,407 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,407 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,407 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:54,407 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:54,407 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:54,407 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:54,408 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:54,408 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:54,408 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:54,408 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:54,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:54,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:54,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:54,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:54,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:54,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:54,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:54,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:54,410 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:54,410 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:54,410 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:54,410 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:54,410 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:54,411 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,411 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:54,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,411 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:54,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:54,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,412 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,414 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:54,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:54,414 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:54,414 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:54,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:54,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:54,415 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,415 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:54,415 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:54,415 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:54,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:54,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:54,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:54,415 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,415 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,415 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,415 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:54,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:54,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:54,415 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:54,415 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:54,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:54,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:54,515 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:54,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:54,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:54,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:54,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:54,518 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:54,518 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:54,518 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:54,518 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:54,519 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,519 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:54,520 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:54,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:54,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:54,520 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,520 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,521 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:54,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:54,521 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:54,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:54,521 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:54,521 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:54,521 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:54,522 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:54,522 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:54,522 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:54,522 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:54,522 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:54,522 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:54,522 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:54,522 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:54,522 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:54,522 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:54,522 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:54,522 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,522 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:54,522 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:54,523 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,523 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:54,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:54,523 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,523 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:54,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:54,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:54,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:54,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:54,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:54,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a617d67[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:54,525 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:54,525 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,525 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,526 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:54,526 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:54,526 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:54,526 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,526 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:54,526 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:54,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:54,526 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:54,527 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:54,527 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:54,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:54,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:54,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:54,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:54,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:54,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:54,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:54,530 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:54,530 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:54,530 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:54,530 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:54,530 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:54,544 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,545 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,545 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:54,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:54,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:54,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:54,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:54,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:54,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:54,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:54,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:54,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:54,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:54,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,550 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,551 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,553 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,553 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:54,553 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:54,553 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,553 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,553 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:54,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:54,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:54,553 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,553 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:54,554 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,554 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:54,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:54,554 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:54,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:54,554 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:54,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:54,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:54,654 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:54,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:54,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:54,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:54,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:54,656 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:54,656 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:54,656 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:54,656 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:54,657 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,657 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:54,657 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:54,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:54,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:54,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:54,658 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,658 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,659 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,659 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:54,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:54,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:54,659 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:54,659 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:54,659 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:54,659 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:54,659 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:54,659 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:54,660 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:54,660 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:54,660 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:54,660 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:54,660 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:54,660 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:54,660 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:54,660 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:54,660 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,660 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:54,660 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:54,661 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,661 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:54,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:54,661 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,661 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:54,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:54,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:54,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:54,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:54,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:54,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70ba54e8[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:54,663 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:54,663 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,663 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,663 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:54,663 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,664 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:54,663 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:54,664 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:54,665 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:54,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:54,666 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:54,666 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:54,666 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:54,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:54,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:54,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:54,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:54,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:54,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:54,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:54,668 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:54,668 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:54,668 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:54,668 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:54,668 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:54,669 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,669 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:54,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:54,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,669 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:54,669 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:54,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:54,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:54,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:54,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:54,670 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:54,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:54,671 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:54,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:54,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40560 deadline: 1681995354669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,672 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,672 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:54,672 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:54,672 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:54,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:54,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:54,672 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:54,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:54,672 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:54,672 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:54,672 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:54,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:54,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:54,672 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:54,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:54,672 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,673 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,673 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,771 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=572 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/86fcb359ac984e04a3fb97b07d712977 2023-04-20 12:54:54,772 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:54,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:54,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:54,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:54,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:54,778 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:54,778 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:54,778 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:54,778 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:54,778 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/42e016c4a8914e9b9b45f35c5d793427 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/42e016c4a8914e9b9b45f35c5d793427 2023-04-20 12:54:54,778 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,778 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:54,779 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:54,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:54,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:54,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:54,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:54,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:54,780 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:54,781 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:54,781 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:54,781 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:54,781 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:54,781 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:54,783 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/42e016c4a8914e9b9b45f35c5d793427, entries=200, sequenceid=572, filesize=14.3 K 2023-04-20 12:54:54,784 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/12edb73e88ab4c92aec1de9a76a5a56a as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/12edb73e88ab4c92aec1de9a76a5a56a 2023-04-20 12:54:54,785 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:54,785 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:54,785 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:54,785 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,785 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:54,785 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:54,785 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:54,786 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:54,786 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:54,786 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,786 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:54,786 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,786 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:54,786 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:54,788 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/12edb73e88ab4c92aec1de9a76a5a56a, entries=150, sequenceid=572, filesize=11.9 K 2023-04-20 12:54:54,789 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/86fcb359ac984e04a3fb97b07d712977 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/86fcb359ac984e04a3fb97b07d712977 2023-04-20 12:54:54,794 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,795 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:54,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:54,795 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,795 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:54,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:54,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:54,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:54,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:54,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:54,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32daec8d[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:54,796 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:54,796 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,799 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,799 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:54,799 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:54,799 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:54,799 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,799 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:54,800 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:54,800 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,800 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:54,800 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:54,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:54,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:54,801 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/86fcb359ac984e04a3fb97b07d712977, entries=150, sequenceid=572, filesize=11.9 K 2023-04-20 12:54:54,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:54,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:54,802 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 969c768384097479d79029eb29913551 in 1329ms, sequenceid=572, compaction requested=true 2023-04-20 12:54:54,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:54,802 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:54,802 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:54,802 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:54:54,802 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:54:54,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,802 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:54:54,803 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:54:54,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:54,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:54,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,804 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39952 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:54:54,804 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): 969c768384097479d79029eb29913551/A is initiating minor compaction (all files) 2023-04-20 12:54:54,804 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37512 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:54:54,804 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 969c768384097479d79029eb29913551/A in TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:54,804 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): 969c768384097479d79029eb29913551/B is initiating minor compaction (all files) 2023-04-20 12:54:54,804 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/0673a82ad94449fb8c9e6f990cedea6b, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/234cc413c8c24ee193712af21a20a2fa, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/42e016c4a8914e9b9b45f35c5d793427] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp, totalSize=39.0 K 2023-04-20 12:54:54,804 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 969c768384097479d79029eb29913551/B in TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:54,804 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/41f25a07113845e7946debf0069fccd1, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/953a4760cc0046c8acd0ee37e1f290e5, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/12edb73e88ab4c92aec1de9a76a5a56a] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp, totalSize=36.6 K 2023-04-20 12:54:54,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:54,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:54,805 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 41f25a07113845e7946debf0069fccd1, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=521, earliestPutTs=1681995291674 2023-04-20 12:54:54,805 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:54,805 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:54,805 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:54,805 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:54,805 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:54,805 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 0673a82ad94449fb8c9e6f990cedea6b, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=521, earliestPutTs=1681995291674 2023-04-20 12:54:54,806 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 953a4760cc0046c8acd0ee37e1f290e5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=557, earliestPutTs=1681995291753 2023-04-20 12:54:54,806 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 234cc413c8c24ee193712af21a20a2fa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=557, earliestPutTs=1681995291753 2023-04-20 12:54:54,806 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 42e016c4a8914e9b9b45f35c5d793427, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1681995292846 2023-04-20 12:54:54,807 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 12edb73e88ab4c92aec1de9a76a5a56a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1681995292846 2023-04-20 12:54:54,811 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,811 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,811 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:54,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:54,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:54,812 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): 969c768384097479d79029eb29913551#B#compaction#274 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:54,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:54,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:54,812 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:54,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:54,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:54,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:54,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,814 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 969c768384097479d79029eb29913551#A#compaction#275 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:54,814 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:54,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:54,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,819 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:54,819 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:54,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:54,820 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:54,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:54,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:54,820 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:54,820 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:54,820 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:54,820 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,820 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:54,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:54,820 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:54,820 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,820 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:54,820 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:54,920 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:54,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:54,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:54,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:54,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:54,925 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:54,925 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:54,925 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:54,925 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:54,925 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,925 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:54,927 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,927 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:54,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:54,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:54,927 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:54,927 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:54,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:54,927 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:54,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:54,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:54,928 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:54,928 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:54,928 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:54,928 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:54,928 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:54,928 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:54,928 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:54,928 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:54,928 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:54,928 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:54,929 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 969c768384097479d79029eb29913551 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-20 12:54:54,929 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=A 2023-04-20 12:54:54,929 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:54,929 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=B 2023-04-20 12:54:54,929 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:54,929 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=C 2023-04-20 12:54:54,929 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:55,248 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/0b5f24ee4df0411683151ecc12f6ca69 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/0b5f24ee4df0411683151ecc12f6ca69 2023-04-20 12:54:55,252 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/728c11ddfd9a4b9795157a780e5bbbc3 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/728c11ddfd9a4b9795157a780e5bbbc3 2023-04-20 12:54:55,253 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 969c768384097479d79029eb29913551/A of 969c768384097479d79029eb29913551 into 0b5f24ee4df0411683151ecc12f6ca69(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:54:55,253 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:55,253 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551., storeName=969c768384097479d79029eb29913551/A, priority=13, startTime=1681995294802; duration=0sec 2023-04-20 12:54:55,254 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:54:55,254 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:54:55,254 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37512 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:54:55,255 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): 969c768384097479d79029eb29913551/C is initiating minor compaction (all files) 2023-04-20 12:54:55,255 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 969c768384097479d79029eb29913551/C in TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:55,255 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/a420974e5e064667bd2f9c26387afe80, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/b66b4e46a6fe47709a8791c32623d95a, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/86fcb359ac984e04a3fb97b07d712977] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp, totalSize=36.6 K 2023-04-20 12:54:55,255 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting a420974e5e064667bd2f9c26387afe80, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=521, earliestPutTs=1681995291674 2023-04-20 12:54:55,256 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting b66b4e46a6fe47709a8791c32623d95a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=557, earliestPutTs=1681995291753 2023-04-20 12:54:55,256 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 86fcb359ac984e04a3fb97b07d712977, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1681995292846 2023-04-20 12:54:55,258 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 969c768384097479d79029eb29913551/B of 969c768384097479d79029eb29913551 into 728c11ddfd9a4b9795157a780e5bbbc3(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:54:55,258 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:55,258 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551., storeName=969c768384097479d79029eb29913551/B, priority=13, startTime=1681995294802; duration=0sec 2023-04-20 12:54:55,258 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:55,262 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 969c768384097479d79029eb29913551#C#compaction#277 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:55,276 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/7a9dcc8e738943558bf895732d1e4416 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/7a9dcc8e738943558bf895732d1e4416 2023-04-20 12:54:55,279 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 969c768384097479d79029eb29913551/C of 969c768384097479d79029eb29913551 into 7a9dcc8e738943558bf895732d1e4416(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:54:55,280 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:55,280 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551., storeName=969c768384097479d79029eb29913551/C, priority=13, startTime=1681995294803; duration=0sec 2023-04-20 12:54:55,280 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:55,339 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=592 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/7b09d36d92464e0a9f6a3aedbf1c5eac 2023-04-20 12:54:55,352 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=592 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/573132105c6f4aa69bad361af30ff694 2023-04-20 12:54:55,397 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=592 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/915164780d2549528a6aad8dc48aa2c2 2023-04-20 12:54:55,401 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/7b09d36d92464e0a9f6a3aedbf1c5eac as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/7b09d36d92464e0a9f6a3aedbf1c5eac 2023-04-20 12:54:55,405 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/7b09d36d92464e0a9f6a3aedbf1c5eac, entries=150, sequenceid=592, filesize=11.9 K 2023-04-20 12:54:55,406 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/573132105c6f4aa69bad361af30ff694 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/573132105c6f4aa69bad361af30ff694 2023-04-20 12:54:55,411 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/573132105c6f4aa69bad361af30ff694, entries=150, sequenceid=592, filesize=11.9 K 2023-04-20 12:54:55,412 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/915164780d2549528a6aad8dc48aa2c2 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/915164780d2549528a6aad8dc48aa2c2 2023-04-20 12:54:55,432 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/915164780d2549528a6aad8dc48aa2c2, entries=150, sequenceid=592, filesize=11.9 K 2023-04-20 12:54:55,433 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for 969c768384097479d79029eb29913551 in 504ms, sequenceid=592, compaction requested=false 2023-04-20 12:54:55,433 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:55,433 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:55,433 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:55,433 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:55,433 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,433 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:55,433 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:55,434 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:55,434 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:55,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:55,435 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:55,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:55,435 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:55,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:55,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:55,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:55,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:55,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6afbb457[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:55,436 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:55,436 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:55,437 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:55,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:55,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:55,437 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:55,437 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:55,437 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:55,438 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,438 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:55,440 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:55,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:55,441 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:55,441 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:55,441 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:55,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:55,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:55,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:55,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:55,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:55,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:55,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:55,444 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:55,444 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:55,444 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:55,444 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:55,444 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:55,453 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:55,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:55,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:55,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:55,453 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:55,453 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:55,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:55,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:55,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:55,453 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:55,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:55,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:55,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:55,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:55,455 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:55,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:55,466 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:55,466 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:55,466 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:55,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:55,466 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:55,466 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:55,466 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:55,466 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,466 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:55,466 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:55,466 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:55,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:55,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:55,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:55,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:55,467 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:55,467 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:55,467 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:55,567 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:55,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:55,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:55,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:55,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:55,570 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:55,570 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:55,570 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:55,571 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:55,571 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:55,571 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:55,572 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:55,572 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:55,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:55,572 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,572 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:55,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:55,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:55,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:55,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:55,573 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:55,573 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:55,573 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:55,573 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:55,573 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:55,573 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:55,574 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:55,574 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:55,574 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:55,574 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:55,574 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:55,574 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:55,574 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:55,575 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:55,575 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,575 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:55,575 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:55,576 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:55,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:55,576 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:55,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:55,577 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:55,577 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:55,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:55,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:55,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:55,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:55,578 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:55,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b40762a[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:55,578 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:55,579 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:55,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:55,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:55,579 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:55,579 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:55,579 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:55,579 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,579 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:55,580 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:55,580 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,580 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:55,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:55,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:55,580 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:55,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:55,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:55,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:55,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:55,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:55,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:55,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:55,582 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:55,582 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:55,582 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:55,582 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:55,582 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:55,583 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:55,583 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:55,583 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:55,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:55,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:55,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:55,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:55,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:55,584 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:55,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:55,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:55,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:55,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:55,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:55,585 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:55,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:55,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:55,588 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:55,588 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:55,588 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:55,588 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:55,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:55,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:55,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:55,588 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:55,588 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:55,588 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,588 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:55,588 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:55,588 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:55,588 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:55,588 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:55,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:55,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:55,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:55,689 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:55,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:55,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:55,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:55,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:55,694 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:55,694 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:55,694 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:55,694 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:55,694 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:55,694 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:55,695 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,695 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:55,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:55,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:55,695 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,695 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:55,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:55,696 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:55,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:55,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:55,696 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:55,698 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:55,698 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:55,698 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:55,698 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:55,698 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:55,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 969c768384097479d79029eb29913551 2023-04-20 12:54:55,700 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 969c768384097479d79029eb29913551 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-20 12:54:55,700 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=A 2023-04-20 12:54:55,700 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:55,700 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=B 2023-04-20 12:54:55,700 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:55,700 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=C 2023-04-20 12:54:55,700 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:55,701 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:55,701 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:55,701 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:55,701 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:55,701 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:55,701 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:55,701 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:55,702 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:55,702 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,702 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:55,702 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:55,711 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,711 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:55,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:55,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:55,712 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:55,712 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:55,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:55,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:55,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:55,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:55,713 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=606 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/b67f2e2b5ff246dc8d41f08d2e8cbd60 2023-04-20 12:54:55,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:55,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79ff74be[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:55,713 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:55,714 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:55,716 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:55,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:55,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:55,716 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:55,716 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:55,716 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:55,716 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,716 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:55,717 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:55,717 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,717 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:55,718 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:55,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:55,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:55,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:55,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:55,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:55,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:55,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:55,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:55,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:55,720 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:55,720 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:55,720 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:55,720 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:55,720 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:55,722 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,723 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,724 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:55,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:55,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:55,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:55,724 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:55,724 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:55,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:55,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:55,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:55,725 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:55,725 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,725 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:55,725 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:55,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:55,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:55,725 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:55,725 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,725 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:55,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:55,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:55,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:55,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:55,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:55,726 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:55,726 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:55,725 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:55,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:55,726 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:55,726 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:55,727 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:55,757 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681995295747, max timestamp=1681995295756Num uniques 859; 2023-04-20 12:54:55,757 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 12:54:55,758 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681995295747, max timestamp=1681995295756Num uniques 859; 2023-04-20 12:54:55,758 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681995295747, max timestamp=1681995295756Num uniques 859; 2023-04-20 12:54:55,758 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 12:54:55,758 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 12:54:55,814 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:55,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 350 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40560 deadline: 1681995355811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,826 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:55,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:55,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:55,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:55,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:55,829 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:55,829 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:55,829 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:55,829 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:55,829 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:55,829 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:55,830 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:55,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:55,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:55,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:55,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:55,831 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:55,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:55,831 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:55,831 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,831 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:55,831 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:55,831 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:55,831 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:55,831 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:55,831 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:55,831 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:55,831 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:55,831 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:55,831 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:55,831 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:55,832 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:55,832 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:55,832 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:55,832 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,832 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:55,832 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:55,833 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,833 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:55,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:55,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:55,834 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:55,834 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:55,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:55,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:55,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:55,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:55,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:55,835 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:55,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ca45188[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:55,835 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:55,837 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:55,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:55,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:55,837 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:55,837 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:55,837 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:55,837 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,837 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:55,838 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:55,838 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:55,838 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:55,838 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:55,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:55,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:55,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:55,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:55,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:55,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:55,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:55,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:55,840 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:55,840 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:55,840 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:55,840 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:55,840 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:55,841 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:55,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:55,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:55,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:55,841 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:55,841 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:55,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:55,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:55,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:55,842 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:55,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:55,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:55,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:55,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:55,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:55,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,843 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:55,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:55,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,844 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,844 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:55,844 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:55,844 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:55,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:55,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:55,844 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,844 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:55,844 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:55,844 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:55,845 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:55,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:55,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:55,845 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:55,845 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:55,845 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:55,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:55,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:55,916 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:55,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 352 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40560 deadline: 1681995355915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,919 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:55,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40554 deadline: 1681995355916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,921 DEBUG [Thread-1496] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4145 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551., hostname=jenkins-hbase20.apache.org,36277,1681995213947, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-20 12:54:55,945 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:55,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:55,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:55,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:55,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:55,949 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:55,949 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:55,949 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:55,949 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:55,950 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:55,950 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:55,950 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,950 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:55,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:55,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:55,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:55,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:55,951 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,951 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:55,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:55,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:55,951 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:55,953 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:55,953 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:55,953 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:55,953 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:55,953 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:55,954 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:55,954 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:55,954 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:55,954 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:55,954 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:55,954 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:55,954 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:55,954 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:55,954 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,954 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:55,954 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:55,969 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,969 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:55,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:55,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:55,970 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:55,970 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:55,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:55,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:55,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:55,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:55,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:55,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b4a0747[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:55,971 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:55,971 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:55,972 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:55,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:55,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:55,972 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:55,972 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:55,972 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:55,972 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,973 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:55,973 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:55,973 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:55,973 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:55,973 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:55,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:55,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:55,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:55,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:55,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:55,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:55,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:55,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:55,977 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:55,977 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:55,977 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:55,977 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:55,977 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:55,978 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:55,978 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:55,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:55,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:55,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:55,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:55,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:55,978 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:55,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:55,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:55,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:55,979 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:55,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:55,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:55,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:55,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,981 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:55,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:55,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,982 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,982 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:55,986 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:55,986 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:55,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:55,986 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:55,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:55,986 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:55,986 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:55,986 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:55,986 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:55,986 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:55,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:55,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:55,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:55,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:55,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:55,987 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:55,987 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:56,087 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:56,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:56,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:56,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:56,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:56,090 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:56,090 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:56,090 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:56,090 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:56,090 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:56,090 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:56,091 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:56,091 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:56,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:56,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:56,091 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:56,091 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:56,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:56,091 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:56,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:56,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:56,092 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:56,092 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:56,092 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:56,092 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:56,092 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:56,092 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:56,092 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:56,092 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:56,092 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:56,092 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:56,092 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:56,092 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:56,092 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:56,092 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:56,092 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:56,092 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:56,092 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:56,093 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:56,093 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:56,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:56,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:56,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:56,093 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:56,093 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:56,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:56,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:56,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:56,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:56,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:56,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:56,095 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:56,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73e3c522[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:56,095 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:56,095 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:56,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:56,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:56,095 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:56,095 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:56,095 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:56,095 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:56,095 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:56,096 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:56,096 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:56,096 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:56,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:56,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:56,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:56,096 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:56,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:56,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:56,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:56,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:56,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:56,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:56,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:56,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:56,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:56,098 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:56,098 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:56,098 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:56,098 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:56,098 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:56,099 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:56,099 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:56,099 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:56,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:56,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:56,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:56,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:56,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:56,100 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:56,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:56,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:56,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:56,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:56,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:56,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:56,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:56,101 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:56,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:56,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:56,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:56,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:56,102 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:56,102 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:56,102 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:56,102 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:56,102 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:56,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:56,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:56,102 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:56,102 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:56,102 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:56,102 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:56,102 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:56,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:56,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:56,102 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:56,103 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:56,102 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:56,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:56,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:56,119 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:56,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 354 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40560 deadline: 1681995356117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:56,143 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=606 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/8d0437155587436b9e9c18bb0ee64ea7 2023-04-20 12:54:56,152 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=606 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/5a669b69fb9f42f282b2a16d9c94712c 2023-04-20 12:54:56,157 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/b67f2e2b5ff246dc8d41f08d2e8cbd60 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/b67f2e2b5ff246dc8d41f08d2e8cbd60 2023-04-20 12:54:56,160 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/b67f2e2b5ff246dc8d41f08d2e8cbd60, entries=150, sequenceid=606, filesize=11.9 K 2023-04-20 12:54:56,161 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/8d0437155587436b9e9c18bb0ee64ea7 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/8d0437155587436b9e9c18bb0ee64ea7 2023-04-20 12:54:56,164 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/8d0437155587436b9e9c18bb0ee64ea7, entries=150, sequenceid=606, filesize=11.9 K 2023-04-20 12:54:56,164 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/5a669b69fb9f42f282b2a16d9c94712c as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/5a669b69fb9f42f282b2a16d9c94712c 2023-04-20 12:54:56,169 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/5a669b69fb9f42f282b2a16d9c94712c, entries=150, sequenceid=606, filesize=11.9 K 2023-04-20 12:54:56,169 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 969c768384097479d79029eb29913551 in 469ms, sequenceid=606, compaction requested=true 2023-04-20 12:54:56,170 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:56,170 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:56,170 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:54:56,170 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:54:56,170 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:54:56,170 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:54:56,172 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37614 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:54:56,172 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): 969c768384097479d79029eb29913551/B is initiating minor compaction (all files) 2023-04-20 12:54:56,172 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37614 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:54:56,172 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 969c768384097479d79029eb29913551/B in TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:56,172 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): 969c768384097479d79029eb29913551/A is initiating minor compaction (all files) 2023-04-20 12:54:56,173 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/728c11ddfd9a4b9795157a780e5bbbc3, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/573132105c6f4aa69bad361af30ff694, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/8d0437155587436b9e9c18bb0ee64ea7] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp, totalSize=36.7 K 2023-04-20 12:54:56,173 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 969c768384097479d79029eb29913551/A in TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:56,173 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/0b5f24ee4df0411683151ecc12f6ca69, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/7b09d36d92464e0a9f6a3aedbf1c5eac, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/b67f2e2b5ff246dc8d41f08d2e8cbd60] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp, totalSize=36.7 K 2023-04-20 12:54:56,173 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 728c11ddfd9a4b9795157a780e5bbbc3, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1681995292846 2023-04-20 12:54:56,173 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 0b5f24ee4df0411683151ecc12f6ca69, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1681995292846 2023-04-20 12:54:56,173 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 573132105c6f4aa69bad361af30ff694, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=592, earliestPutTs=1681995293527 2023-04-20 12:54:56,173 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 7b09d36d92464e0a9f6a3aedbf1c5eac, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=592, earliestPutTs=1681995293527 2023-04-20 12:54:56,174 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 8d0437155587436b9e9c18bb0ee64ea7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=606, earliestPutTs=1681995295691 2023-04-20 12:54:56,174 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting b67f2e2b5ff246dc8d41f08d2e8cbd60, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=606, earliestPutTs=1681995295691 2023-04-20 12:54:56,181 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): 969c768384097479d79029eb29913551#B#compaction#283 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:56,183 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 969c768384097479d79029eb29913551#A#compaction#284 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:56,203 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:56,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:56,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:56,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:56,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:56,207 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:56,207 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:56,207 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:56,207 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:56,207 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:56,207 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:56,208 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:56,208 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:56,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:56,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:56,208 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:56,208 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:56,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:56,209 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:56,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:56,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:56,209 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:56,210 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:56,210 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:56,210 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:56,210 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:56,210 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:56,210 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:56,210 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:56,210 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:56,210 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 969c768384097479d79029eb29913551 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-20 12:54:56,211 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=A 2023-04-20 12:54:56,211 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:54:56,211 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=B 2023-04-20 12:54:56,211 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:54:56,211 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=C 2023-04-20 12:54:56,211 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:54:56,221 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/f16be217161b4137abfabd209781da3c as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/f16be217161b4137abfabd209781da3c 2023-04-20 12:54:56,233 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/7c26b249eb4e4203b68946c00c89befc as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/7c26b249eb4e4203b68946c00c89befc 2023-04-20 12:54:56,233 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 969c768384097479d79029eb29913551/A of 969c768384097479d79029eb29913551 into f16be217161b4137abfabd209781da3c(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:54:56,233 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:56,233 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551., storeName=969c768384097479d79029eb29913551/A, priority=13, startTime=1681995296170; duration=0sec 2023-04-20 12:54:56,233 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:54:56,233 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:54:56,236 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37614 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:54:56,236 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): 969c768384097479d79029eb29913551/C is initiating minor compaction (all files) 2023-04-20 12:54:56,236 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 969c768384097479d79029eb29913551/C in TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:56,236 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/7a9dcc8e738943558bf895732d1e4416, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/915164780d2549528a6aad8dc48aa2c2, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/5a669b69fb9f42f282b2a16d9c94712c] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp, totalSize=36.7 K 2023-04-20 12:54:56,236 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 7a9dcc8e738943558bf895732d1e4416, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1681995292846 2023-04-20 12:54:56,237 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 915164780d2549528a6aad8dc48aa2c2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=592, earliestPutTs=1681995293527 2023-04-20 12:54:56,237 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 5a669b69fb9f42f282b2a16d9c94712c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=606, earliestPutTs=1681995295691 2023-04-20 12:54:56,239 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 969c768384097479d79029eb29913551/B of 969c768384097479d79029eb29913551 into 7c26b249eb4e4203b68946c00c89befc(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:54:56,239 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:56,239 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551., storeName=969c768384097479d79029eb29913551/B, priority=13, startTime=1681995296170; duration=0sec 2023-04-20 12:54:56,239 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:56,244 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 969c768384097479d79029eb29913551#C#compaction#286 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:56,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 969c768384097479d79029eb29913551 2023-04-20 12:54:56,424 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:56,442 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:56,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 363 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40560 deadline: 1681995356440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:56,544 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:56,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 365 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40560 deadline: 1681995356543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:56,644 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=643 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/51c6c25d71ab45d7a35912eeebcdc18e 2023-04-20 12:54:56,657 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/82c4eaed416841d881f18c348a1e0a07 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/82c4eaed416841d881f18c348a1e0a07 2023-04-20 12:54:56,662 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 969c768384097479d79029eb29913551/C of 969c768384097479d79029eb29913551 into 82c4eaed416841d881f18c348a1e0a07(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:54:56,662 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:56,662 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551., storeName=969c768384097479d79029eb29913551/C, priority=13, startTime=1681995296170; duration=0sec 2023-04-20 12:54:56,662 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:56,696 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:56,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40616 deadline: 1681995356694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:56,698 DEBUG [Thread-1494] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18167 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551., hostname=jenkins-hbase20.apache.org,36277,1681995213947, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-20 12:54:56,725 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:56,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40570 deadline: 1681995356724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:56,727 DEBUG [Thread-1488] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18193 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551., hostname=jenkins-hbase20.apache.org,36277,1681995213947, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-20 12:54:56,747 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:56,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 367 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40560 deadline: 1681995356746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:56,797 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:56,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40568 deadline: 1681995356796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:56,798 DEBUG [Thread-1492] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18266 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551., hostname=jenkins-hbase20.apache.org,36277,1681995213947, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-20 12:54:57,049 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:57,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 369 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40560 deadline: 1681995357048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,052 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=643 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/58ea8f6666a847bab6e17cddf3532442 2023-04-20 12:54:57,460 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=643 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/4009c9f92d774c47a6dcf7d18d8649d2 2023-04-20 12:54:57,463 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/51c6c25d71ab45d7a35912eeebcdc18e as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/51c6c25d71ab45d7a35912eeebcdc18e 2023-04-20 12:54:57,467 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/51c6c25d71ab45d7a35912eeebcdc18e, entries=150, sequenceid=643, filesize=11.9 K 2023-04-20 12:54:57,467 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/58ea8f6666a847bab6e17cddf3532442 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/58ea8f6666a847bab6e17cddf3532442 2023-04-20 12:54:57,470 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/58ea8f6666a847bab6e17cddf3532442, entries=150, sequenceid=643, filesize=11.9 K 2023-04-20 12:54:57,471 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/4009c9f92d774c47a6dcf7d18d8649d2 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/4009c9f92d774c47a6dcf7d18d8649d2 2023-04-20 12:54:57,474 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/4009c9f92d774c47a6dcf7d18d8649d2, entries=150, sequenceid=643, filesize=11.9 K 2023-04-20 12:54:57,475 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 969c768384097479d79029eb29913551 in 1265ms, sequenceid=643, compaction requested=false 2023-04-20 12:54:57,475 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:57,475 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:57,475 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:57,475 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:57,475 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,475 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:57,475 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:57,476 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,476 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:57,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:57,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:57,476 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:57,476 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:57,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:57,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:57,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:57,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:57,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:57,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71bb0564[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:57,480 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:57,480 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:57,491 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:57,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:57,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:57,491 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:57,491 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:57,491 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:57,491 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,491 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:57,494 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:57,494 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:57,495 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:57,495 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:57,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:57,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:57,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:57,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:57,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:57,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:57,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:57,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:57,498 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:57,498 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:57,498 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:57,498 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:57,498 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:57,503 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:57,503 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:57,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:57,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:57,503 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:57,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:57,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:57,503 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:57,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:57,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:57,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:57,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:57,505 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:57,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:57,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:57,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,507 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:57,507 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:57,507 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:57,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:57,507 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:57,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:57,507 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:57,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:57,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:57,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:57,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:57,508 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:57,508 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,508 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:57,508 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:57,508 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:57,508 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:57,508 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:57,509 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:57,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 969c768384097479d79029eb29913551 2023-04-20 12:54:57,553 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 969c768384097479d79029eb29913551 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-20 12:54:57,553 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=A 2023-04-20 12:54:57,553 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:57,553 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=B 2023-04-20 12:54:57,553 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:57,553 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=C 2023-04-20 12:54:57,553 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:54:57,564 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=657 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/01a3ad8a878d49d6a64f717dee476114 2023-04-20 12:54:57,579 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=657 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/4a8462683d5c49f8ab2c37d389e840bf 2023-04-20 12:54:57,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681995297594, max timestamp=1681995297606Num uniques 859; 2023-04-20 12:54:57,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681995297594, max timestamp=1681995297606Num uniques 859; 2023-04-20 12:54:57,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681995297594, max timestamp=1681995297606Num uniques 859; 2023-04-20 12:54:57,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 12:54:57,608 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:57,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 12:54:57,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 12:54:57,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:57,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:57,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:57,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:57,612 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:57,612 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:57,612 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:57,612 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:57,612 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:57,612 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:57,613 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,613 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:57,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:57,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:57,613 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,613 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:57,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:57,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:57,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:57,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:57,614 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:57,615 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:57,615 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:57,615 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:57,615 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:57,615 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:57,618 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:57,618 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:57,618 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:57,618 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:57,618 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:57,619 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:57,619 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:57,619 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:57,619 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,619 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:57,619 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:57,628 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:57,628 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:57,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:57,628 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:57,628 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:57,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:57,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:57,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:57,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:57,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:57,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fad22ca[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:57,630 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:57,630 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:57,632 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:57,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:57,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:57,632 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:57,633 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:57,633 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:57,632 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,633 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:57,633 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,633 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:57,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:57,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:57,633 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:57,634 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:57,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:57,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:57,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:57,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:57,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:57,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:57,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:57,636 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:57,636 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:57,636 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:57,636 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:57,636 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:57,640 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:57,640 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:57,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:57,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:57,641 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:57,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:57,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:57,642 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,644 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:57,644 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:57,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:57,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:57,644 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:57,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:57,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:57,645 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:57,645 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:57,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:57,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:57,645 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:57,645 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:57,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:57,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:57,645 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:57,645 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:57,645 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,645 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:57,645 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:57,645 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:57,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:57,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:57,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:57,646 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:57,670 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:57,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 405 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40560 deadline: 1681995357668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,745 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:57,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:57,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:57,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:57,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:57,749 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:57,749 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:57,749 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:57,749 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:57,749 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:57,749 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:57,750 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:57,750 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:57,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:57,750 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,750 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:57,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:57,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:57,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:57,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:57,750 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:57,751 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:57,751 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:57,751 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:57,751 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:57,751 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:57,754 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:57,754 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:57,754 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:57,754 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:57,754 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:57,754 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:57,754 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:57,754 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:57,754 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,754 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:57,754 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:57,755 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,755 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:57,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:57,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:57,755 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:57,755 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:57,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:57,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:57,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:57,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:57,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:57,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34267447[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:57,757 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:57,757 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:57,757 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:57,758 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,758 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:57,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:57,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:57,758 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:57,758 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:57,758 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:57,759 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:57,759 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:57,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:57,759 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:57,759 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:57,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:57,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:57,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:57,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:57,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:57,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:57,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:57,761 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:57,761 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:57,761 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:57,761 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:57,761 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:57,763 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:57,763 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:57,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:57,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:57,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:57,763 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:57,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:57,763 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:57,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:57,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:57,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:57,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:57,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:57,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:57,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:57,764 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:57,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:57,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,765 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,765 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:57,765 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:57,765 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,765 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:57,765 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:57,765 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:57,766 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:54:57,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:57,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:57,766 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:57,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:57,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:57,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:57,765 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:57,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:57,766 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:57,766 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:57,766 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:57,774 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:57,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 407 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40560 deadline: 1681995357771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,866 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:57,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:57,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:54:57,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:54:57,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:54:57,869 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:54:57,869 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:57,869 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:54:57,869 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:54:57,870 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:57,870 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:57,870 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,870 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:57,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:57,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:57,871 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,871 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:54:57,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:57,871 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:57,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:54:57,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:57,871 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:54:57,871 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:54:57,871 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:54:57,872 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:54:57,872 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:54:57,872 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:54:57,872 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:54:57,872 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:54:57,872 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:57,872 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. started... 2023-04-20 12:54:57,872 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. as already flushing 2023-04-20 12:54:57,872 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:57,872 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:54:57,872 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:54:57,872 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,872 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:54:57,872 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:54:57,873 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,873 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:57,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:57,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:57,873 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:57,873 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:54:57,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:57,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:57,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:57,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:57,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:54:57,874 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:54:57,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59016d34[Count = 0] remaining members to acquire global barrier 2023-04-20 12:54:57,874 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:57,875 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:57,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:57,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:57,875 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:54:57,875 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:54:57,875 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:54:57,875 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,875 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:54:57,876 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,876 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:54:57,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:57,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:57,876 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:57,876 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:54:57,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:57,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:57,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:57,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:57,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:57,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:54:57,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:54:57,878 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:54:57,878 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:54:57,878 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:54:57,878 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:54:57,878 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:54:57,880 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:57,880 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:57,880 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:57,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:57,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:54:57,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:54:57,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:57,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:57,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:57,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:57,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:54:57,881 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:57,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:57,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:54:57,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:57,882 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:54:57,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:54:57,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,891 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,891 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:54:57,891 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:57,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:54:57,891 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:54:57,891 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:54:57,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:54:57,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:54:57,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:54:57,891 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:54:57,891 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,891 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:57,891 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:54:57,891 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:54:57,891 DEBUG [Thread-1498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:54:57,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:54:57,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:54:57,891 DEBUG [Thread-1498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:54:57,928 DEBUG [Thread-1507] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x38204078 to 127.0.0.1:56673 2023-04-20 12:54:57,928 DEBUG [Thread-1507] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:54:57,928 DEBUG [Thread-1501] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x353e11a8 to 127.0.0.1:56673 2023-04-20 12:54:57,929 DEBUG [Thread-1501] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:54:57,930 DEBUG [Thread-1503] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x12f583c6 to 127.0.0.1:56673 2023-04-20 12:54:57,930 DEBUG [Thread-1503] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:54:57,930 DEBUG [Thread-1505] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6635f17a to 127.0.0.1:56673 2023-04-20 12:54:57,930 DEBUG [Thread-1505] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:54:57,931 DEBUG [Thread-1499] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x07742ca6 to 127.0.0.1:56673 2023-04-20 12:54:57,931 DEBUG [Thread-1499] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:54:57,977 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:54:57,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 409 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40560 deadline: 1681995357977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:54:57,991 DEBUG [Thread-1498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:54:57,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:54:57,993 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=657 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/bb13d012879b4e77a05b2231825ef698 2023-04-20 12:54:58,001 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/01a3ad8a878d49d6a64f717dee476114 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/01a3ad8a878d49d6a64f717dee476114 2023-04-20 12:54:58,008 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/01a3ad8a878d49d6a64f717dee476114, entries=150, sequenceid=657, filesize=11.9 K 2023-04-20 12:54:58,009 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/4a8462683d5c49f8ab2c37d389e840bf as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/4a8462683d5c49f8ab2c37d389e840bf 2023-04-20 12:54:58,014 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/4a8462683d5c49f8ab2c37d389e840bf, entries=150, sequenceid=657, filesize=11.9 K 2023-04-20 12:54:58,015 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/bb13d012879b4e77a05b2231825ef698 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/bb13d012879b4e77a05b2231825ef698 2023-04-20 12:54:58,021 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/bb13d012879b4e77a05b2231825ef698, entries=150, sequenceid=657, filesize=11.9 K 2023-04-20 12:54:58,022 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 969c768384097479d79029eb29913551 in 469ms, sequenceid=657, compaction requested=true 2023-04-20 12:54:58,023 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:58,023 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:58,023 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:54:58,023 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:54:58,023 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:54:58,023 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:54:58,024 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37716 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:54:58,024 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): 969c768384097479d79029eb29913551/A is initiating minor compaction (all files) 2023-04-20 12:54:58,024 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37716 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:54:58,024 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 969c768384097479d79029eb29913551/A in TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:58,025 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): 969c768384097479d79029eb29913551/B is initiating minor compaction (all files) 2023-04-20 12:54:58,025 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/f16be217161b4137abfabd209781da3c, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/51c6c25d71ab45d7a35912eeebcdc18e, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/01a3ad8a878d49d6a64f717dee476114] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp, totalSize=36.8 K 2023-04-20 12:54:58,025 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 969c768384097479d79029eb29913551/B in TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:58,025 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/7c26b249eb4e4203b68946c00c89befc, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/58ea8f6666a847bab6e17cddf3532442, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/4a8462683d5c49f8ab2c37d389e840bf] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp, totalSize=36.8 K 2023-04-20 12:54:58,025 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting f16be217161b4137abfabd209781da3c, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=606, earliestPutTs=1681995295691 2023-04-20 12:54:58,025 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 7c26b249eb4e4203b68946c00c89befc, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=606, earliestPutTs=1681995295691 2023-04-20 12:54:58,025 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 51c6c25d71ab45d7a35912eeebcdc18e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=643, earliestPutTs=1681995295795 2023-04-20 12:54:58,026 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 58ea8f6666a847bab6e17cddf3532442, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=643, earliestPutTs=1681995295795 2023-04-20 12:54:58,026 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 01a3ad8a878d49d6a64f717dee476114, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=657, earliestPutTs=1681995296433 2023-04-20 12:54:58,026 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 4a8462683d5c49f8ab2c37d389e840bf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=657, earliestPutTs=1681995296433 2023-04-20 12:54:58,033 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): 969c768384097479d79029eb29913551#B#compaction#292 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:58,035 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 969c768384097479d79029eb29913551#A#compaction#293 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:58,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 969c768384097479d79029eb29913551 2023-04-20 12:54:58,280 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 969c768384097479d79029eb29913551 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-20 12:54:58,280 DEBUG [Thread-1490] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5f2b45f8 to 127.0.0.1:56673 2023-04-20 12:54:58,281 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=A 2023-04-20 12:54:58,281 DEBUG [Thread-1490] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:54:58,281 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:54:58,281 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=B 2023-04-20 12:54:58,281 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:54:58,281 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=C 2023-04-20 12:54:58,281 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:54:58,447 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/9bfcd65502ec45918a2cc9bfdae967ae as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/9bfcd65502ec45918a2cc9bfdae967ae 2023-04-20 12:54:58,448 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/11024e6847c741bc83bc9f17e02ed849 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/11024e6847c741bc83bc9f17e02ed849 2023-04-20 12:54:58,453 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 969c768384097479d79029eb29913551/B of 969c768384097479d79029eb29913551 into 9bfcd65502ec45918a2cc9bfdae967ae(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:54:58,453 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:58,453 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551., storeName=969c768384097479d79029eb29913551/B, priority=13, startTime=1681995298023; duration=0sec 2023-04-20 12:54:58,453 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:54:58,453 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:54:58,453 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 969c768384097479d79029eb29913551/A of 969c768384097479d79029eb29913551 into 11024e6847c741bc83bc9f17e02ed849(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:54:58,454 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:58,454 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551., storeName=969c768384097479d79029eb29913551/A, priority=13, startTime=1681995298023; duration=0sec 2023-04-20 12:54:58,454 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:58,454 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37716 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:54:58,455 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): 969c768384097479d79029eb29913551/C is initiating minor compaction (all files) 2023-04-20 12:54:58,455 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 969c768384097479d79029eb29913551/C in TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:54:58,455 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/82c4eaed416841d881f18c348a1e0a07, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/4009c9f92d774c47a6dcf7d18d8649d2, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/bb13d012879b4e77a05b2231825ef698] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp, totalSize=36.8 K 2023-04-20 12:54:58,455 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 82c4eaed416841d881f18c348a1e0a07, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=606, earliestPutTs=1681995295691 2023-04-20 12:54:58,455 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 4009c9f92d774c47a6dcf7d18d8649d2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=643, earliestPutTs=1681995295795 2023-04-20 12:54:58,456 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting bb13d012879b4e77a05b2231825ef698, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=657, earliestPutTs=1681995296433 2023-04-20 12:54:58,463 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): 969c768384097479d79029eb29913551#C#compaction#295 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:54:58,688 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=694 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/f0e7290ab372419f89c4b27d1821e474 2023-04-20 12:54:58,879 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/8c96726f1e6a42a7ad84ba3f539f8d4e as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/8c96726f1e6a42a7ad84ba3f539f8d4e 2023-04-20 12:54:58,883 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 969c768384097479d79029eb29913551/C of 969c768384097479d79029eb29913551 into 8c96726f1e6a42a7ad84ba3f539f8d4e(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:54:58,883 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:58,883 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551., storeName=969c768384097479d79029eb29913551/C, priority=13, startTime=1681995298023; duration=0sec 2023-04-20 12:54:58,883 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:54:59,108 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=694 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/661c389ff58648b68bea15d79087871e 2023-04-20 12:54:59,528 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=694 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/e7005d105fa84046be8c0895cc6353ed 2023-04-20 12:54:59,532 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/f0e7290ab372419f89c4b27d1821e474 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/f0e7290ab372419f89c4b27d1821e474 2023-04-20 12:54:59,537 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/f0e7290ab372419f89c4b27d1821e474, entries=150, sequenceid=694, filesize=11.9 K 2023-04-20 12:54:59,537 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/661c389ff58648b68bea15d79087871e as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/661c389ff58648b68bea15d79087871e 2023-04-20 12:54:59,541 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/661c389ff58648b68bea15d79087871e, entries=150, sequenceid=694, filesize=11.9 K 2023-04-20 12:54:59,542 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/e7005d105fa84046be8c0895cc6353ed as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/e7005d105fa84046be8c0895cc6353ed 2023-04-20 12:54:59,545 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/e7005d105fa84046be8c0895cc6353ed, entries=150, sequenceid=694, filesize=11.9 K 2023-04-20 12:54:59,546 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=0 B/0 for 969c768384097479d79029eb29913551 in 1266ms, sequenceid=694, compaction requested=false 2023-04-20 12:54:59,546 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 969c768384097479d79029eb29913551: 2023-04-20 12:54:59,953 DEBUG [Thread-1496] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5ffcf146 to 127.0.0.1:56673 2023-04-20 12:54:59,953 DEBUG [Thread-1496] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:55:06,782 DEBUG [Thread-1494] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7e9921a7 to 127.0.0.1:56673 2023-04-20 12:55:06,782 DEBUG [Thread-1494] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:55:06,793 DEBUG [Thread-1488] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2135e03b to 127.0.0.1:56673 2023-04-20 12:55:06,793 DEBUG [Thread-1488] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:55:06,873 DEBUG [Thread-1492] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x24e1e878 to 127.0.0.1:56673 2023-04-20 12:55:06,873 DEBUG [Thread-1492] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:55:06,873 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-20 12:55:06,873 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(394): wrote 41 2023-04-20 12:55:06,873 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(394): wrote 280 2023-04-20 12:55:06,874 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(394): wrote 35 2023-04-20 12:55:06,874 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(394): wrote 35 2023-04-20 12:55:06,874 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(394): wrote 182 2023-04-20 12:55:06,874 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-20 12:55:06,874 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(398): read 6004 2023-04-20 12:55:06,874 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(398): read 5829 2023-04-20 12:55:06,874 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(398): read 5776 2023-04-20 12:55:06,874 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(398): read 6057 2023-04-20 12:55:06,874 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(398): read 5785 2023-04-20 12:55:06,874 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-20 12:55:06,875 INFO [Listener at localhost.localdomain/42205] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-20 12:55:06,875 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x73ef029c to 127.0.0.1:56673 2023-04-20 12:55:06,875 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:55:06,876 INFO [Listener at localhost.localdomain/42205] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-20 12:55:06,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-20 12:55:06,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-20 12:55:06,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-20 12:55:06,882 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681995306882"}]},"ts":"1681995306882"} 2023-04-20 12:55:06,885 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-20 12:55:06,890 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-20 12:55:06,891 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=969c768384097479d79029eb29913551, UNASSIGN}] 2023-04-20 12:55:06,892 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=969c768384097479d79029eb29913551, UNASSIGN 2023-04-20 12:55:06,893 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=969c768384097479d79029eb29913551, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:06,893 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681995306893"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681995306893"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681995306893"}]},"ts":"1681995306893"} 2023-04-20 12:55:06,894 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure 969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947}] 2023-04-20 12:55:06,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-20 12:55:07,047 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 969c768384097479d79029eb29913551 2023-04-20 12:55:07,047 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 969c768384097479d79029eb29913551, disabling compactions & flushes 2023-04-20 12:55:07,047 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:55:07,047 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:55:07,047 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. after waiting 0 ms 2023-04-20 12:55:07,047 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:55:07,047 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 969c768384097479d79029eb29913551 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-20 12:55:07,048 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=A 2023-04-20 12:55:07,048 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:07,048 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=B 2023-04-20 12:55:07,048 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:07,049 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 969c768384097479d79029eb29913551, store=C 2023-04-20 12:55:07,049 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:07,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-20 12:55:07,462 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=704 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/be2fb27a05134dd89f6c62b1afcfe0a9 2023-04-20 12:55:07,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-20 12:55:07,878 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=704 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/2868505a27e24480b5a071a4a0616119 2023-04-20 12:55:07,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-20 12:55:08,288 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=704 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/28d78d23fb5249b083906976393b570e 2023-04-20 12:55:08,292 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/A/be2fb27a05134dd89f6c62b1afcfe0a9 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/be2fb27a05134dd89f6c62b1afcfe0a9 2023-04-20 12:55:08,301 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/be2fb27a05134dd89f6c62b1afcfe0a9, entries=150, sequenceid=704, filesize=11.9 K 2023-04-20 12:55:08,302 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/B/2868505a27e24480b5a071a4a0616119 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/2868505a27e24480b5a071a4a0616119 2023-04-20 12:55:08,306 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/2868505a27e24480b5a071a4a0616119, entries=150, sequenceid=704, filesize=11.9 K 2023-04-20 12:55:08,307 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/.tmp/C/28d78d23fb5249b083906976393b570e as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/28d78d23fb5249b083906976393b570e 2023-04-20 12:55:08,311 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/28d78d23fb5249b083906976393b570e, entries=150, sequenceid=704, filesize=11.9 K 2023-04-20 12:55:08,311 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 969c768384097479d79029eb29913551 in 1264ms, sequenceid=704, compaction requested=true 2023-04-20 12:55:08,334 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/2ec5f1d068c749fab513bba61c5aaa9b, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/ee09fad504084976853fc119bbbebf14, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/09f3cd45a0ca4c6badd8ed43ab362fef, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/edafa87cf153417081ded0dc9e2d9133, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/ec66b7c5cfe14c33aee80948edb46eeb, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/a826d7ab90394257ade3dd452c0e8f9a, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/aa1a05911dc44fe79800e60ba76790c7, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/00512d487f20498ab21ea438e7665422, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/5818b8b12bfa4563a389a62871f9d4f1, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/0daed446938d4efd8813719b59573423, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/7f9679f37bd1408099826e5df79a3cda, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/787cf09f785e44aebb8713509bb90fcc, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/fb7d5472239745a2948c6a428f08261f, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/405898cfc63f4079bc7a91ea24d1eecd, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/7dab9f1c1b404e38a3c8dac8a9e3cf87, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/c7a2705510f04c15872fb2ab06bb7a45, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/d3a6e3f8afbb403399b06633fb598d27, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/3a7a3e8c0b2940a6896586d35b481942, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/b1e672be0ab24c2e932b7d0e02e0133a, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/29d0ee9899d34e0099d4a7746913f829, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/27e82770a2fb4bdba87e4f2a551cb15c, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/832784cb368a493baf2129a852536900, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/a9e39d2252ac43d68277aa7c7828ceca, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/3a1629a43bd94c74bee8893e8e6dff1e, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/2d35bed43ed64ca0bf23cbeab53ce7f1, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/9cb53b653ed54598b808171c0e93c6f5, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/ad15889485ae4001967c0b1d401a496e, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/0673a82ad94449fb8c9e6f990cedea6b, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/da06d81b86eb44a4b4afca9fa862bc34, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/234cc413c8c24ee193712af21a20a2fa, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/42e016c4a8914e9b9b45f35c5d793427, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/0b5f24ee4df0411683151ecc12f6ca69, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/7b09d36d92464e0a9f6a3aedbf1c5eac, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/f16be217161b4137abfabd209781da3c, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/b67f2e2b5ff246dc8d41f08d2e8cbd60, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/51c6c25d71ab45d7a35912eeebcdc18e, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/01a3ad8a878d49d6a64f717dee476114] to archive 2023-04-20 12:55:08,334 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-20 12:55:08,336 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/2ec5f1d068c749fab513bba61c5aaa9b to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/2ec5f1d068c749fab513bba61c5aaa9b 2023-04-20 12:55:08,337 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/ee09fad504084976853fc119bbbebf14 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/ee09fad504084976853fc119bbbebf14 2023-04-20 12:55:08,338 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/09f3cd45a0ca4c6badd8ed43ab362fef to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/09f3cd45a0ca4c6badd8ed43ab362fef 2023-04-20 12:55:08,340 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/edafa87cf153417081ded0dc9e2d9133 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/edafa87cf153417081ded0dc9e2d9133 2023-04-20 12:55:08,341 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/ec66b7c5cfe14c33aee80948edb46eeb to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/ec66b7c5cfe14c33aee80948edb46eeb 2023-04-20 12:55:08,342 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/a826d7ab90394257ade3dd452c0e8f9a to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/a826d7ab90394257ade3dd452c0e8f9a 2023-04-20 12:55:08,343 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/aa1a05911dc44fe79800e60ba76790c7 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/aa1a05911dc44fe79800e60ba76790c7 2023-04-20 12:55:08,344 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/00512d487f20498ab21ea438e7665422 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/00512d487f20498ab21ea438e7665422 2023-04-20 12:55:08,345 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/5818b8b12bfa4563a389a62871f9d4f1 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/5818b8b12bfa4563a389a62871f9d4f1 2023-04-20 12:55:08,346 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/0daed446938d4efd8813719b59573423 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/0daed446938d4efd8813719b59573423 2023-04-20 12:55:08,347 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/7f9679f37bd1408099826e5df79a3cda to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/7f9679f37bd1408099826e5df79a3cda 2023-04-20 12:55:08,347 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/787cf09f785e44aebb8713509bb90fcc to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/787cf09f785e44aebb8713509bb90fcc 2023-04-20 12:55:08,348 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/fb7d5472239745a2948c6a428f08261f to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/fb7d5472239745a2948c6a428f08261f 2023-04-20 12:55:08,349 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/405898cfc63f4079bc7a91ea24d1eecd to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/405898cfc63f4079bc7a91ea24d1eecd 2023-04-20 12:55:08,350 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/7dab9f1c1b404e38a3c8dac8a9e3cf87 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/7dab9f1c1b404e38a3c8dac8a9e3cf87 2023-04-20 12:55:08,351 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/c7a2705510f04c15872fb2ab06bb7a45 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/c7a2705510f04c15872fb2ab06bb7a45 2023-04-20 12:55:08,352 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/d3a6e3f8afbb403399b06633fb598d27 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/d3a6e3f8afbb403399b06633fb598d27 2023-04-20 12:55:08,353 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/3a7a3e8c0b2940a6896586d35b481942 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/3a7a3e8c0b2940a6896586d35b481942 2023-04-20 12:55:08,355 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/b1e672be0ab24c2e932b7d0e02e0133a to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/b1e672be0ab24c2e932b7d0e02e0133a 2023-04-20 12:55:08,356 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/29d0ee9899d34e0099d4a7746913f829 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/29d0ee9899d34e0099d4a7746913f829 2023-04-20 12:55:08,357 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/27e82770a2fb4bdba87e4f2a551cb15c to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/27e82770a2fb4bdba87e4f2a551cb15c 2023-04-20 12:55:08,358 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/832784cb368a493baf2129a852536900 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/832784cb368a493baf2129a852536900 2023-04-20 12:55:08,359 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/a9e39d2252ac43d68277aa7c7828ceca to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/a9e39d2252ac43d68277aa7c7828ceca 2023-04-20 12:55:08,360 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/3a1629a43bd94c74bee8893e8e6dff1e to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/3a1629a43bd94c74bee8893e8e6dff1e 2023-04-20 12:55:08,360 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/2d35bed43ed64ca0bf23cbeab53ce7f1 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/2d35bed43ed64ca0bf23cbeab53ce7f1 2023-04-20 12:55:08,361 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/9cb53b653ed54598b808171c0e93c6f5 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/9cb53b653ed54598b808171c0e93c6f5 2023-04-20 12:55:08,362 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/ad15889485ae4001967c0b1d401a496e to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/ad15889485ae4001967c0b1d401a496e 2023-04-20 12:55:08,363 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/0673a82ad94449fb8c9e6f990cedea6b to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/0673a82ad94449fb8c9e6f990cedea6b 2023-04-20 12:55:08,364 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/da06d81b86eb44a4b4afca9fa862bc34 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/da06d81b86eb44a4b4afca9fa862bc34 2023-04-20 12:55:08,365 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/234cc413c8c24ee193712af21a20a2fa to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/234cc413c8c24ee193712af21a20a2fa 2023-04-20 12:55:08,366 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/42e016c4a8914e9b9b45f35c5d793427 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/42e016c4a8914e9b9b45f35c5d793427 2023-04-20 12:55:08,367 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/0b5f24ee4df0411683151ecc12f6ca69 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/0b5f24ee4df0411683151ecc12f6ca69 2023-04-20 12:55:08,368 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/7b09d36d92464e0a9f6a3aedbf1c5eac to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/7b09d36d92464e0a9f6a3aedbf1c5eac 2023-04-20 12:55:08,369 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/f16be217161b4137abfabd209781da3c to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/f16be217161b4137abfabd209781da3c 2023-04-20 12:55:08,370 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/b67f2e2b5ff246dc8d41f08d2e8cbd60 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/b67f2e2b5ff246dc8d41f08d2e8cbd60 2023-04-20 12:55:08,371 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/51c6c25d71ab45d7a35912eeebcdc18e to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/51c6c25d71ab45d7a35912eeebcdc18e 2023-04-20 12:55:08,372 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/01a3ad8a878d49d6a64f717dee476114 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/01a3ad8a878d49d6a64f717dee476114 2023-04-20 12:55:08,375 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/41beeff62f1c4175be854898a83d1cbd, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/7d4e295258d34a15a1d9ff50d2b65ad4, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/813b983bf05c425086c808bf1972f72d, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/0da123b9a3ff48bc8ba39fd7aedd686f, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/42d0d7bec070428a8b59b5914c0dab9c, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/9c97fc731af945ee90e24f2f21754a5b, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/41caef2949f140c18e523a83e5ba863e, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/701bfda29614462a95ba3bf61fd418b8, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/2735ca4481314a9e935dbd7c3529f5cb, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/6c31345bb55f4528a8557d551e36fda9, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/d654be6e03524d82802a4211c680416f, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/e9b3a9aef53f4a4aa0115103db81d446, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/1fae6d9869334991bc22b2b65d3e9628, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/b02c8cbd1b15444c852c01a1731b76f6, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/11de23e5dcbb4ed0bebc97267f64ec43, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/c0052b67c89c48609a808267bebcad22, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/37a874107a7d49c392ec4f162e7233a8, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/8c04bf595cb54203a35d56d7b6585511, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/1ad6e3d840c14525b6423b948ed3efbf, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/6752be952a5b4e309061525c71d0a106, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/577d6977dc2f47eea5af5a7d98974f5e, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/b6a27d4d936f4a41a7d415a3ef4fdd6a, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/8b6401c6284a41b6a399dee240007481, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/b85ff8ccd07f42688d57336fdcbbb58f, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/d9fa90dbd864445c9e92049307912677, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/a816b0a0a45648188ef6932137a5dbe3, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/7022386c71304ee2be60687beba29aac, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/41f25a07113845e7946debf0069fccd1, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/2cc6856eeece4a1081bf9bca5a103c6c, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/953a4760cc0046c8acd0ee37e1f290e5, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/728c11ddfd9a4b9795157a780e5bbbc3, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/12edb73e88ab4c92aec1de9a76a5a56a, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/573132105c6f4aa69bad361af30ff694, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/7c26b249eb4e4203b68946c00c89befc, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/8d0437155587436b9e9c18bb0ee64ea7, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/58ea8f6666a847bab6e17cddf3532442, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/4a8462683d5c49f8ab2c37d389e840bf] to archive 2023-04-20 12:55:08,376 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-20 12:55:08,378 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/41beeff62f1c4175be854898a83d1cbd to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/41beeff62f1c4175be854898a83d1cbd 2023-04-20 12:55:08,379 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/7d4e295258d34a15a1d9ff50d2b65ad4 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/7d4e295258d34a15a1d9ff50d2b65ad4 2023-04-20 12:55:08,380 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/813b983bf05c425086c808bf1972f72d to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/813b983bf05c425086c808bf1972f72d 2023-04-20 12:55:08,381 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/0da123b9a3ff48bc8ba39fd7aedd686f to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/0da123b9a3ff48bc8ba39fd7aedd686f 2023-04-20 12:55:08,382 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/42d0d7bec070428a8b59b5914c0dab9c to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/42d0d7bec070428a8b59b5914c0dab9c 2023-04-20 12:55:08,383 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/9c97fc731af945ee90e24f2f21754a5b to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/9c97fc731af945ee90e24f2f21754a5b 2023-04-20 12:55:08,384 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/41caef2949f140c18e523a83e5ba863e to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/41caef2949f140c18e523a83e5ba863e 2023-04-20 12:55:08,384 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/701bfda29614462a95ba3bf61fd418b8 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/701bfda29614462a95ba3bf61fd418b8 2023-04-20 12:55:08,385 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/2735ca4481314a9e935dbd7c3529f5cb to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/2735ca4481314a9e935dbd7c3529f5cb 2023-04-20 12:55:08,386 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/6c31345bb55f4528a8557d551e36fda9 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/6c31345bb55f4528a8557d551e36fda9 2023-04-20 12:55:08,386 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/d654be6e03524d82802a4211c680416f to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/d654be6e03524d82802a4211c680416f 2023-04-20 12:55:08,387 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/e9b3a9aef53f4a4aa0115103db81d446 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/e9b3a9aef53f4a4aa0115103db81d446 2023-04-20 12:55:08,388 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/1fae6d9869334991bc22b2b65d3e9628 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/1fae6d9869334991bc22b2b65d3e9628 2023-04-20 12:55:08,389 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/b02c8cbd1b15444c852c01a1731b76f6 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/b02c8cbd1b15444c852c01a1731b76f6 2023-04-20 12:55:08,390 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/11de23e5dcbb4ed0bebc97267f64ec43 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/11de23e5dcbb4ed0bebc97267f64ec43 2023-04-20 12:55:08,391 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/c0052b67c89c48609a808267bebcad22 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/c0052b67c89c48609a808267bebcad22 2023-04-20 12:55:08,391 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/37a874107a7d49c392ec4f162e7233a8 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/37a874107a7d49c392ec4f162e7233a8 2023-04-20 12:55:08,392 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/8c04bf595cb54203a35d56d7b6585511 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/8c04bf595cb54203a35d56d7b6585511 2023-04-20 12:55:08,393 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/1ad6e3d840c14525b6423b948ed3efbf to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/1ad6e3d840c14525b6423b948ed3efbf 2023-04-20 12:55:08,394 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/6752be952a5b4e309061525c71d0a106 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/6752be952a5b4e309061525c71d0a106 2023-04-20 12:55:08,395 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/577d6977dc2f47eea5af5a7d98974f5e to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/577d6977dc2f47eea5af5a7d98974f5e 2023-04-20 12:55:08,395 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/b6a27d4d936f4a41a7d415a3ef4fdd6a to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/b6a27d4d936f4a41a7d415a3ef4fdd6a 2023-04-20 12:55:08,396 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/8b6401c6284a41b6a399dee240007481 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/8b6401c6284a41b6a399dee240007481 2023-04-20 12:55:08,397 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/b85ff8ccd07f42688d57336fdcbbb58f to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/b85ff8ccd07f42688d57336fdcbbb58f 2023-04-20 12:55:08,398 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/d9fa90dbd864445c9e92049307912677 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/d9fa90dbd864445c9e92049307912677 2023-04-20 12:55:08,398 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/a816b0a0a45648188ef6932137a5dbe3 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/a816b0a0a45648188ef6932137a5dbe3 2023-04-20 12:55:08,399 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/7022386c71304ee2be60687beba29aac to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/7022386c71304ee2be60687beba29aac 2023-04-20 12:55:08,400 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/41f25a07113845e7946debf0069fccd1 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/41f25a07113845e7946debf0069fccd1 2023-04-20 12:55:08,400 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/2cc6856eeece4a1081bf9bca5a103c6c to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/2cc6856eeece4a1081bf9bca5a103c6c 2023-04-20 12:55:08,401 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/953a4760cc0046c8acd0ee37e1f290e5 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/953a4760cc0046c8acd0ee37e1f290e5 2023-04-20 12:55:08,402 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/728c11ddfd9a4b9795157a780e5bbbc3 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/728c11ddfd9a4b9795157a780e5bbbc3 2023-04-20 12:55:08,403 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/12edb73e88ab4c92aec1de9a76a5a56a to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/12edb73e88ab4c92aec1de9a76a5a56a 2023-04-20 12:55:08,403 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/573132105c6f4aa69bad361af30ff694 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/573132105c6f4aa69bad361af30ff694 2023-04-20 12:55:08,404 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/7c26b249eb4e4203b68946c00c89befc to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/7c26b249eb4e4203b68946c00c89befc 2023-04-20 12:55:08,405 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/8d0437155587436b9e9c18bb0ee64ea7 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/8d0437155587436b9e9c18bb0ee64ea7 2023-04-20 12:55:08,406 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/58ea8f6666a847bab6e17cddf3532442 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/58ea8f6666a847bab6e17cddf3532442 2023-04-20 12:55:08,406 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/4a8462683d5c49f8ab2c37d389e840bf to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/4a8462683d5c49f8ab2c37d389e840bf 2023-04-20 12:55:08,410 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/56045c8f78ab4a9aa1fc117ae886e1e8, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/9d3c20565923423fbc69c1099c659957, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/f83daecfe8664c7b867a8091f1b32de6, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/843d81ff59044f368098fc1f4b075949, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/cc0d805fe21747de9414a8fdfe621d31, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/ce233aa0bbad4d9387ba032d6c7b099a, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/a552cd724c7344ff8a423c633b647784, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/a0fa60a555ac49169fdf71c91157dbc0, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/f1379ca7ad0f4f28a8ca4b59192cd475, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/73e8627ea7e249e9808aad7cd18906e6, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/00250837fd23488c908cec25c82873f4, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/fe16225f2fdc48b9a234d58f47783efe, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/3f890e3f20cb43eeb1fc61a555fdb610, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/6a9004bf92904432b67bb5cd65ce8c14, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/3f68a3f4c12446db9846b95a839e35b8, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/a874bad3e90a468887728c67b5ed7808, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/5ecc31bcb3c64e3c94eacf34ee997ebe, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/bfae4ce02339452faa94eaa462bc031c, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/856a9d3640994a898b7e528c560f38e2, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/0a77acd8cdd1474489e5416eaf89fd5d, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/b731c58d0de249258e2a7dca947a08a1, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/5e70ec1697cf41548a6ba4854633dccc, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/79f282fd60674c50a588a5433f0a79ef, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/63947960e67c49d1a4e0c6033c92dcdf, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/9b4e8e73339c4d34bd5a7d2d164f1f3b, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/0df0aae0a86046e6bdb42a9404784d2b, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/5f7d56760e5644b387db282ec2de5143, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/a420974e5e064667bd2f9c26387afe80, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/12181e248643468a92f079c4fc66498c, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/b66b4e46a6fe47709a8791c32623d95a, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/7a9dcc8e738943558bf895732d1e4416, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/86fcb359ac984e04a3fb97b07d712977, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/915164780d2549528a6aad8dc48aa2c2, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/82c4eaed416841d881f18c348a1e0a07, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/5a669b69fb9f42f282b2a16d9c94712c, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/4009c9f92d774c47a6dcf7d18d8649d2, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/bb13d012879b4e77a05b2231825ef698] to archive 2023-04-20 12:55:08,411 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-20 12:55:08,412 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/56045c8f78ab4a9aa1fc117ae886e1e8 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/56045c8f78ab4a9aa1fc117ae886e1e8 2023-04-20 12:55:08,413 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/9d3c20565923423fbc69c1099c659957 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/9d3c20565923423fbc69c1099c659957 2023-04-20 12:55:08,414 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/f83daecfe8664c7b867a8091f1b32de6 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/f83daecfe8664c7b867a8091f1b32de6 2023-04-20 12:55:08,415 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/843d81ff59044f368098fc1f4b075949 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/843d81ff59044f368098fc1f4b075949 2023-04-20 12:55:08,415 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/cc0d805fe21747de9414a8fdfe621d31 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/cc0d805fe21747de9414a8fdfe621d31 2023-04-20 12:55:08,416 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/ce233aa0bbad4d9387ba032d6c7b099a to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/ce233aa0bbad4d9387ba032d6c7b099a 2023-04-20 12:55:08,417 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/a552cd724c7344ff8a423c633b647784 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/a552cd724c7344ff8a423c633b647784 2023-04-20 12:55:08,418 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/a0fa60a555ac49169fdf71c91157dbc0 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/a0fa60a555ac49169fdf71c91157dbc0 2023-04-20 12:55:08,419 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/f1379ca7ad0f4f28a8ca4b59192cd475 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/f1379ca7ad0f4f28a8ca4b59192cd475 2023-04-20 12:55:08,420 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/73e8627ea7e249e9808aad7cd18906e6 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/73e8627ea7e249e9808aad7cd18906e6 2023-04-20 12:55:08,421 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/00250837fd23488c908cec25c82873f4 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/00250837fd23488c908cec25c82873f4 2023-04-20 12:55:08,422 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/fe16225f2fdc48b9a234d58f47783efe to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/fe16225f2fdc48b9a234d58f47783efe 2023-04-20 12:55:08,423 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/3f890e3f20cb43eeb1fc61a555fdb610 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/3f890e3f20cb43eeb1fc61a555fdb610 2023-04-20 12:55:08,424 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/6a9004bf92904432b67bb5cd65ce8c14 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/6a9004bf92904432b67bb5cd65ce8c14 2023-04-20 12:55:08,425 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/3f68a3f4c12446db9846b95a839e35b8 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/3f68a3f4c12446db9846b95a839e35b8 2023-04-20 12:55:08,426 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/a874bad3e90a468887728c67b5ed7808 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/a874bad3e90a468887728c67b5ed7808 2023-04-20 12:55:08,427 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/5ecc31bcb3c64e3c94eacf34ee997ebe to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/5ecc31bcb3c64e3c94eacf34ee997ebe 2023-04-20 12:55:08,428 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/bfae4ce02339452faa94eaa462bc031c to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/bfae4ce02339452faa94eaa462bc031c 2023-04-20 12:55:08,429 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/856a9d3640994a898b7e528c560f38e2 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/856a9d3640994a898b7e528c560f38e2 2023-04-20 12:55:08,429 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/0a77acd8cdd1474489e5416eaf89fd5d to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/0a77acd8cdd1474489e5416eaf89fd5d 2023-04-20 12:55:08,430 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/b731c58d0de249258e2a7dca947a08a1 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/b731c58d0de249258e2a7dca947a08a1 2023-04-20 12:55:08,431 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/5e70ec1697cf41548a6ba4854633dccc to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/5e70ec1697cf41548a6ba4854633dccc 2023-04-20 12:55:08,432 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/79f282fd60674c50a588a5433f0a79ef to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/79f282fd60674c50a588a5433f0a79ef 2023-04-20 12:55:08,433 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/63947960e67c49d1a4e0c6033c92dcdf to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/63947960e67c49d1a4e0c6033c92dcdf 2023-04-20 12:55:08,433 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/9b4e8e73339c4d34bd5a7d2d164f1f3b to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/9b4e8e73339c4d34bd5a7d2d164f1f3b 2023-04-20 12:55:08,434 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/0df0aae0a86046e6bdb42a9404784d2b to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/0df0aae0a86046e6bdb42a9404784d2b 2023-04-20 12:55:08,435 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/5f7d56760e5644b387db282ec2de5143 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/5f7d56760e5644b387db282ec2de5143 2023-04-20 12:55:08,436 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/a420974e5e064667bd2f9c26387afe80 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/a420974e5e064667bd2f9c26387afe80 2023-04-20 12:55:08,437 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/12181e248643468a92f079c4fc66498c to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/12181e248643468a92f079c4fc66498c 2023-04-20 12:55:08,437 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/b66b4e46a6fe47709a8791c32623d95a to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/b66b4e46a6fe47709a8791c32623d95a 2023-04-20 12:55:08,438 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/7a9dcc8e738943558bf895732d1e4416 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/7a9dcc8e738943558bf895732d1e4416 2023-04-20 12:55:08,439 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/86fcb359ac984e04a3fb97b07d712977 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/86fcb359ac984e04a3fb97b07d712977 2023-04-20 12:55:08,439 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/915164780d2549528a6aad8dc48aa2c2 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/915164780d2549528a6aad8dc48aa2c2 2023-04-20 12:55:08,440 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/82c4eaed416841d881f18c348a1e0a07 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/82c4eaed416841d881f18c348a1e0a07 2023-04-20 12:55:08,441 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/5a669b69fb9f42f282b2a16d9c94712c to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/5a669b69fb9f42f282b2a16d9c94712c 2023-04-20 12:55:08,442 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/4009c9f92d774c47a6dcf7d18d8649d2 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/4009c9f92d774c47a6dcf7d18d8649d2 2023-04-20 12:55:08,442 DEBUG [StoreCloser-TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/bb13d012879b4e77a05b2231825ef698 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/bb13d012879b4e77a05b2231825ef698 2023-04-20 12:55:08,445 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/recovered.edits/707.seqid, newMaxSeqId=707, maxSeqId=1 2023-04-20 12:55:08,447 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551. 2023-04-20 12:55:08,447 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 969c768384097479d79029eb29913551: 2023-04-20 12:55:08,448 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 969c768384097479d79029eb29913551 2023-04-20 12:55:08,448 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=969c768384097479d79029eb29913551, regionState=CLOSED 2023-04-20 12:55:08,448 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681995308448"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681995308448"}]},"ts":"1681995308448"} 2023-04-20 12:55:08,450 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-04-20 12:55:08,450 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure 969c768384097479d79029eb29913551, server=jenkins-hbase20.apache.org,36277,1681995213947 in 1.5550 sec 2023-04-20 12:55:08,451 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-04-20 12:55:08,451 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=969c768384097479d79029eb29913551, UNASSIGN in 1.5590 sec 2023-04-20 12:55:08,452 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681995308452"}]},"ts":"1681995308452"} 2023-04-20 12:55:08,452 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-20 12:55:08,453 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-20 12:55:08,454 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5770 sec 2023-04-20 12:55:08,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-20 12:55:08,988 INFO [Listener at localhost.localdomain/42205] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-04-20 12:55:08,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-20 12:55:08,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 12:55:08,995 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 12:55:08,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-20 12:55:08,996 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 12:55:09,001 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551 2023-04-20 12:55:09,003 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A, FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B, FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C, FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/recovered.edits] 2023-04-20 12:55:09,006 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/11024e6847c741bc83bc9f17e02ed849 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/11024e6847c741bc83bc9f17e02ed849 2023-04-20 12:55:09,008 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/be2fb27a05134dd89f6c62b1afcfe0a9 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/be2fb27a05134dd89f6c62b1afcfe0a9 2023-04-20 12:55:09,010 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/f0e7290ab372419f89c4b27d1821e474 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/A/f0e7290ab372419f89c4b27d1821e474 2023-04-20 12:55:09,012 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/2868505a27e24480b5a071a4a0616119 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/2868505a27e24480b5a071a4a0616119 2023-04-20 12:55:09,013 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/661c389ff58648b68bea15d79087871e to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/661c389ff58648b68bea15d79087871e 2023-04-20 12:55:09,015 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/9bfcd65502ec45918a2cc9bfdae967ae to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/B/9bfcd65502ec45918a2cc9bfdae967ae 2023-04-20 12:55:09,018 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/28d78d23fb5249b083906976393b570e to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/28d78d23fb5249b083906976393b570e 2023-04-20 12:55:09,019 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/8c96726f1e6a42a7ad84ba3f539f8d4e to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/8c96726f1e6a42a7ad84ba3f539f8d4e 2023-04-20 12:55:09,020 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/e7005d105fa84046be8c0895cc6353ed to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/C/e7005d105fa84046be8c0895cc6353ed 2023-04-20 12:55:09,021 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/recovered.edits/707.seqid to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551/recovered.edits/707.seqid 2023-04-20 12:55:09,022 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/969c768384097479d79029eb29913551 2023-04-20 12:55:09,022 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-20 12:55:09,023 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 12:55:09,024 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-20 12:55:09,025 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-20 12:55:09,026 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 12:55:09,026 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-20 12:55:09,026 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681995309026"}]},"ts":"9223372036854775807"} 2023-04-20 12:55:09,027 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-20 12:55:09,027 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 969c768384097479d79029eb29913551, NAME => 'TestAcidGuarantees,,1681995275724.969c768384097479d79029eb29913551.', STARTKEY => '', ENDKEY => ''}] 2023-04-20 12:55:09,027 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-20 12:55:09,028 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681995309027"}]},"ts":"9223372036854775807"} 2023-04-20 12:55:09,029 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-20 12:55:09,029 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 12:55:09,030 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 40 msec 2023-04-20 12:55:09,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-20 12:55:09,098 INFO [Listener at localhost.localdomain/42205] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-04-20 12:55:09,116 INFO [Listener at localhost.localdomain/42205] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=224 (was 229), OpenFileDescriptor=537 (was 523) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=314 (was 322), ProcessCount=170 (was 170), AvailableMemoryMB=9164 (was 9230) 2023-04-20 12:55:09,130 INFO [Listener at localhost.localdomain/42205] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=224, OpenFileDescriptor=537, MaxFileDescriptor=60000, SystemLoadAverage=314, ProcessCount=170, AvailableMemoryMB=9164 2023-04-20 12:55:09,132 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-20 12:55:09,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-20 12:55:09,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-20 12:55:09,135 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-20 12:55:09,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-04-20 12:55:09,136 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-20 12:55:09,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-20 12:55:09,137 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09 2023-04-20 12:55:09,138 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09 empty. 2023-04-20 12:55:09,139 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09 2023-04-20 12:55:09,139 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-20 12:55:09,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-20 12:55:09,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-20 12:55:09,550 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-20 12:55:09,555 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => aeecfead619437b2701aee4eba741c09, NAME => 'TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp 2023-04-20 12:55:09,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-20 12:55:09,965 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-20 12:55:09,966 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing aeecfead619437b2701aee4eba741c09, disabling compactions & flushes 2023-04-20 12:55:09,966 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:09,966 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:09,966 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. after waiting 0 ms 2023-04-20 12:55:09,966 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:09,966 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:09,966 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for aeecfead619437b2701aee4eba741c09: 2023-04-20 12:55:09,968 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-20 12:55:09,969 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681995309969"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681995309969"}]},"ts":"1681995309969"} 2023-04-20 12:55:09,970 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-20 12:55:09,970 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-20 12:55:09,970 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681995309970"}]},"ts":"1681995309970"} 2023-04-20 12:55:09,971 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-20 12:55:09,973 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=aeecfead619437b2701aee4eba741c09, ASSIGN}] 2023-04-20 12:55:09,975 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=aeecfead619437b2701aee4eba741c09, ASSIGN 2023-04-20 12:55:09,975 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=aeecfead619437b2701aee4eba741c09, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,36277,1681995213947; forceNewPlan=false, retain=false 2023-04-20 12:55:10,127 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=aeecfead619437b2701aee4eba741c09, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:10,127 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681995310127"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681995310127"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681995310127"}]},"ts":"1681995310127"} 2023-04-20 12:55:10,130 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947}] 2023-04-20 12:55:10,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-20 12:55:10,286 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:10,286 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => aeecfead619437b2701aee4eba741c09, NAME => 'TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09.', STARTKEY => '', ENDKEY => ''} 2023-04-20 12:55:10,286 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees aeecfead619437b2701aee4eba741c09 2023-04-20 12:55:10,287 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-20 12:55:10,287 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for aeecfead619437b2701aee4eba741c09 2023-04-20 12:55:10,287 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for aeecfead619437b2701aee4eba741c09 2023-04-20 12:55:10,288 INFO [StoreOpener-aeecfead619437b2701aee4eba741c09-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region aeecfead619437b2701aee4eba741c09 2023-04-20 12:55:10,289 DEBUG [StoreOpener-aeecfead619437b2701aee4eba741c09-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A 2023-04-20 12:55:10,289 DEBUG [StoreOpener-aeecfead619437b2701aee4eba741c09-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A 2023-04-20 12:55:10,289 INFO [StoreOpener-aeecfead619437b2701aee4eba741c09-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 12:55:10,289 INFO [StoreOpener-aeecfead619437b2701aee4eba741c09-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region aeecfead619437b2701aee4eba741c09 columnFamilyName A 2023-04-20 12:55:10,289 INFO [StoreOpener-aeecfead619437b2701aee4eba741c09-1] regionserver.HStore(310): Store=aeecfead619437b2701aee4eba741c09/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 12:55:10,290 INFO [StoreOpener-aeecfead619437b2701aee4eba741c09-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region aeecfead619437b2701aee4eba741c09 2023-04-20 12:55:10,291 DEBUG [StoreOpener-aeecfead619437b2701aee4eba741c09-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B 2023-04-20 12:55:10,291 DEBUG [StoreOpener-aeecfead619437b2701aee4eba741c09-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B 2023-04-20 12:55:10,291 INFO [StoreOpener-aeecfead619437b2701aee4eba741c09-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 12:55:10,291 INFO [StoreOpener-aeecfead619437b2701aee4eba741c09-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region aeecfead619437b2701aee4eba741c09 columnFamilyName B 2023-04-20 12:55:10,291 INFO [StoreOpener-aeecfead619437b2701aee4eba741c09-1] regionserver.HStore(310): Store=aeecfead619437b2701aee4eba741c09/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 12:55:10,292 INFO [StoreOpener-aeecfead619437b2701aee4eba741c09-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region aeecfead619437b2701aee4eba741c09 2023-04-20 12:55:10,292 DEBUG [StoreOpener-aeecfead619437b2701aee4eba741c09-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C 2023-04-20 12:55:10,292 DEBUG [StoreOpener-aeecfead619437b2701aee4eba741c09-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C 2023-04-20 12:55:10,292 INFO [StoreOpener-aeecfead619437b2701aee4eba741c09-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 12:55:10,293 INFO [StoreOpener-aeecfead619437b2701aee4eba741c09-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region aeecfead619437b2701aee4eba741c09 columnFamilyName C 2023-04-20 12:55:10,293 INFO [StoreOpener-aeecfead619437b2701aee4eba741c09-1] regionserver.HStore(310): Store=aeecfead619437b2701aee4eba741c09/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 12:55:10,293 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:10,294 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09 2023-04-20 12:55:10,294 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09 2023-04-20 12:55:10,295 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-20 12:55:10,296 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for aeecfead619437b2701aee4eba741c09 2023-04-20 12:55:10,297 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-20 12:55:10,298 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened aeecfead619437b2701aee4eba741c09; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72682631, jitterRate=0.08305560052394867}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-20 12:55:10,298 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for aeecfead619437b2701aee4eba741c09: 2023-04-20 12:55:10,298 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09., pid=37, masterSystemTime=1681995310283 2023-04-20 12:55:10,299 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:10,299 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:10,300 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=aeecfead619437b2701aee4eba741c09, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:10,300 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681995310300"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1681995310300"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681995310300"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681995310300"}]},"ts":"1681995310300"} 2023-04-20 12:55:10,302 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-04-20 12:55:10,302 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 in 171 msec 2023-04-20 12:55:10,303 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-04-20 12:55:10,303 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=aeecfead619437b2701aee4eba741c09, ASSIGN in 329 msec 2023-04-20 12:55:10,303 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-20 12:55:10,303 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681995310303"}]},"ts":"1681995310303"} 2023-04-20 12:55:10,304 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-20 12:55:10,305 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-20 12:55:10,306 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1740 sec 2023-04-20 12:55:11,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-20 12:55:11,245 INFO [Listener at localhost.localdomain/42205] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-04-20 12:55:11,246 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x6a409627 to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:55:11,254 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6a849516, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:55:11,257 DEBUG [Listener at localhost.localdomain/42205] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:55:11,260 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:35818, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:55:11,261 DEBUG [Listener at localhost.localdomain/42205] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-20 12:55:11,262 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:53134, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-20 12:55:11,264 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-20 12:55:11,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.HMaster$12(2454): Client=jenkins//148.251.75.209 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-20 12:55:11,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-20 12:55:11,677 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-20 12:55:11,679 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-20 12:55:11,680 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-20 12:55:11,684 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-20 12:55:11,687 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=aeecfead619437b2701aee4eba741c09, REOPEN/MOVE}] 2023-04-20 12:55:11,687 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=aeecfead619437b2701aee4eba741c09, REOPEN/MOVE 2023-04-20 12:55:11,688 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=aeecfead619437b2701aee4eba741c09, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:11,688 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681995311688"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681995311688"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681995311688"}]},"ts":"1681995311688"} 2023-04-20 12:55:11,689 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947}] 2023-04-20 12:55:11,840 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close aeecfead619437b2701aee4eba741c09 2023-04-20 12:55:11,841 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing aeecfead619437b2701aee4eba741c09, disabling compactions & flushes 2023-04-20 12:55:11,842 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:11,842 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:11,842 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. after waiting 0 ms 2023-04-20 12:55:11,842 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:11,847 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-20 12:55:11,848 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:11,848 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for aeecfead619437b2701aee4eba741c09: 2023-04-20 12:55:11,848 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(3509): Not adding moved region record: aeecfead619437b2701aee4eba741c09 to self. 2023-04-20 12:55:11,850 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed aeecfead619437b2701aee4eba741c09 2023-04-20 12:55:11,851 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=aeecfead619437b2701aee4eba741c09, regionState=CLOSED 2023-04-20 12:55:11,851 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681995311851"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681995311851"}]},"ts":"1681995311851"} 2023-04-20 12:55:11,856 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-04-20 12:55:11,856 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 in 164 msec 2023-04-20 12:55:11,857 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=aeecfead619437b2701aee4eba741c09, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase20.apache.org,36277,1681995213947; forceNewPlan=false, retain=true 2023-04-20 12:55:12,008 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=aeecfead619437b2701aee4eba741c09, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,008 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681995312008"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681995312008"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681995312008"}]},"ts":"1681995312008"} 2023-04-20 12:55:12,009 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947}] 2023-04-20 12:55:12,166 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:12,166 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => aeecfead619437b2701aee4eba741c09, NAME => 'TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09.', STARTKEY => '', ENDKEY => ''} 2023-04-20 12:55:12,167 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees aeecfead619437b2701aee4eba741c09 2023-04-20 12:55:12,167 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-20 12:55:12,168 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for aeecfead619437b2701aee4eba741c09 2023-04-20 12:55:12,168 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for aeecfead619437b2701aee4eba741c09 2023-04-20 12:55:12,169 INFO [StoreOpener-aeecfead619437b2701aee4eba741c09-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region aeecfead619437b2701aee4eba741c09 2023-04-20 12:55:12,170 DEBUG [StoreOpener-aeecfead619437b2701aee4eba741c09-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A 2023-04-20 12:55:12,170 DEBUG [StoreOpener-aeecfead619437b2701aee4eba741c09-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A 2023-04-20 12:55:12,171 INFO [StoreOpener-aeecfead619437b2701aee4eba741c09-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 12:55:12,171 INFO [StoreOpener-aeecfead619437b2701aee4eba741c09-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region aeecfead619437b2701aee4eba741c09 columnFamilyName A 2023-04-20 12:55:12,173 INFO [StoreOpener-aeecfead619437b2701aee4eba741c09-1] regionserver.HStore(310): Store=aeecfead619437b2701aee4eba741c09/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 12:55:12,173 INFO [StoreOpener-aeecfead619437b2701aee4eba741c09-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region aeecfead619437b2701aee4eba741c09 2023-04-20 12:55:12,174 DEBUG [StoreOpener-aeecfead619437b2701aee4eba741c09-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B 2023-04-20 12:55:12,174 DEBUG [StoreOpener-aeecfead619437b2701aee4eba741c09-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B 2023-04-20 12:55:12,174 INFO [StoreOpener-aeecfead619437b2701aee4eba741c09-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 12:55:12,175 INFO [StoreOpener-aeecfead619437b2701aee4eba741c09-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region aeecfead619437b2701aee4eba741c09 columnFamilyName B 2023-04-20 12:55:12,175 INFO [StoreOpener-aeecfead619437b2701aee4eba741c09-1] regionserver.HStore(310): Store=aeecfead619437b2701aee4eba741c09/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 12:55:12,175 INFO [StoreOpener-aeecfead619437b2701aee4eba741c09-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region aeecfead619437b2701aee4eba741c09 2023-04-20 12:55:12,176 DEBUG [StoreOpener-aeecfead619437b2701aee4eba741c09-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C 2023-04-20 12:55:12,176 DEBUG [StoreOpener-aeecfead619437b2701aee4eba741c09-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C 2023-04-20 12:55:12,176 INFO [StoreOpener-aeecfead619437b2701aee4eba741c09-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 12:55:12,177 INFO [StoreOpener-aeecfead619437b2701aee4eba741c09-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region aeecfead619437b2701aee4eba741c09 columnFamilyName C 2023-04-20 12:55:12,177 INFO [StoreOpener-aeecfead619437b2701aee4eba741c09-1] regionserver.HStore(310): Store=aeecfead619437b2701aee4eba741c09/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 12:55:12,177 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:12,178 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09 2023-04-20 12:55:12,179 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09 2023-04-20 12:55:12,181 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-20 12:55:12,182 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for aeecfead619437b2701aee4eba741c09 2023-04-20 12:55:12,182 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened aeecfead619437b2701aee4eba741c09; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64982698, jitterRate=-0.031682342290878296}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-20 12:55:12,182 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for aeecfead619437b2701aee4eba741c09: 2023-04-20 12:55:12,183 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09., pid=42, masterSystemTime=1681995312161 2023-04-20 12:55:12,184 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:12,184 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:12,184 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=aeecfead619437b2701aee4eba741c09, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,184 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681995312184"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1681995312184"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681995312184"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681995312184"}]},"ts":"1681995312184"} 2023-04-20 12:55:12,186 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-04-20 12:55:12,186 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 in 176 msec 2023-04-20 12:55:12,187 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-04-20 12:55:12,187 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=aeecfead619437b2701aee4eba741c09, REOPEN/MOVE in 499 msec 2023-04-20 12:55:12,189 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-04-20 12:55:12,189 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 504 msec 2023-04-20 12:55:12,191 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 926 msec 2023-04-20 12:55:12,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-04-20 12:55:12,192 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x6f29a773 to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:55:12,198 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1b6e51f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:55:12,199 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x48681e8f to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:55:12,204 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6ad83ace, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:55:12,204 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x342c272f to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:55:12,209 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3cb8db43, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:55:12,209 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x4e90d228 to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:55:12,213 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1bd0918a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:55:12,214 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x1aec6811 to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:55:12,220 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5225f8f6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:55:12,220 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x144047c5 to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:55:12,224 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@695ffff3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:55:12,224 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x525c59d1 to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:55:12,228 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@44b44230, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:55:12,229 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x078ba718 to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:55:12,235 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@37a9f4e9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:55:12,235 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x1747f9fc to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:55:12,240 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4d2e2f6b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:55:12,240 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x3098b23b to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:55:12,245 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@183835cd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:55:12,251 DEBUG [hconnection-0x37f26094-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:55:12,251 DEBUG [hconnection-0x5be033a3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:55:12,254 DEBUG [hconnection-0x1475fae5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:55:12,254 DEBUG [hconnection-0x24228954-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:55:12,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:12,255 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:35832, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:55:12,255 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:35862, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:55:12,258 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:35842, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:55:12,258 DEBUG [hconnection-0x79ab3b15-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:55:12,258 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:35848, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:55:12,258 DEBUG [hconnection-0x16c5835f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:55:12,258 DEBUG [hconnection-0x727699f7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:55:12,261 DEBUG [hconnection-0x55c000ed-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:55:12,262 DEBUG [hconnection-0x5c6cf8b0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:55:12,262 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:35892, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:55:12,262 DEBUG [hconnection-0x2367c76f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:55:12,267 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:35926, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:55:12,274 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:35878, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:55:12,274 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:35886, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:55:12,274 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:35900, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:55:12,274 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:35916, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:55:12,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:12,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:12,275 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:12,275 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:12,275 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:12,275 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:12,275 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:12,275 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:12,290 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,291 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,291 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:12,293 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:12,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:12,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:12,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:12,294 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:12,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:12,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:12,294 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:12,298 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:12,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on aeecfead619437b2701aee4eba741c09 2023-04-20 12:55:12,306 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:12,306 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:12,306 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:12,306 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing aeecfead619437b2701aee4eba741c09 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-20 12:55:12,306 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=A 2023-04-20 12:55:12,306 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:12,311 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:12,312 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=B 2023-04-20 12:55:12,312 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:12,312 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=C 2023-04-20 12:55:12,312 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:12,314 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:12,314 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:12,314 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:12,317 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:12,317 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:12,317 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:12,317 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:12,317 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:12,317 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,317 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:12,317 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:12,318 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,318 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:12,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:12,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:12,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:12,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:12,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:12,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:12,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:12,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70b8c355[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:12,320 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:12,321 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:12,321 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:12,321 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:12,321 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,322 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:12,322 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:12,322 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:12,322 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:12,322 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:12,323 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:12,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:12,324 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:12,324 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:12,324 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:12,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:12,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:12,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:12,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:12,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:12,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:12,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:12,327 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:12,327 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:12,327 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:12,327 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:12,327 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:12,332 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:12,332 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:12,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:12,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:12,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:12,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:12,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:12,332 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:12,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:12,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:12,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:12,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:12,338 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:12,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35892 deadline: 1681995372332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:12,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:12,339 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35842 deadline: 1681995372333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:12,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:12,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:12,340 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:12,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,342 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,342 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:12,342 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:12,342 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,342 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:12,342 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:12,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:12,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:12,342 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:12,342 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:12,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:12,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:12,342 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:12,342 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:12,342 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:12,342 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:12,342 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:12,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:12,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:12,345 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:12,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35886 deadline: 1681995372338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,345 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:12,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35900 deadline: 1681995372338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,345 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:12,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35848 deadline: 1681995372338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,440 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:12,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35892 deadline: 1681995372440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,441 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:12,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35842 deadline: 1681995372440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,442 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:12,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:12,446 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:12,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35900 deadline: 1681995372446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,447 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:12,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35848 deadline: 1681995372446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,448 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:12,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35886 deadline: 1681995372446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:12,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:12,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:12,458 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:12,458 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:12,458 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:12,458 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:12,459 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:12,459 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:12,459 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:12,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:12,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:12,459 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:12,460 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,460 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:12,460 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:12,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:12,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:12,461 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:12,466 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:12,466 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:12,466 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:12,466 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:12,466 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:12,467 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:12,467 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:12,467 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:12,467 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:12,467 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:12,467 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:12,468 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:12,468 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:12,468 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,470 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:12,470 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:12,473 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:12,473 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:12,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:12,474 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:12,474 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:12,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:12,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:12,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:12,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:12,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:12,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56a514fa[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:12,475 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:12,475 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:12,476 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:12,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:12,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:12,476 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:12,476 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:12,476 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:12,476 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,476 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:12,479 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,479 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:12,479 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:12,479 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:12,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:12,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:12,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:12,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:12,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:12,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:12,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:12,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:12,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:12,484 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:12,484 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:12,484 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:12,484 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:12,484 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:12,490 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:12,490 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:12,491 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:12,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:12,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:12,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:12,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:12,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:12,491 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:12,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:12,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:12,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:12,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:12,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:12,492 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:12,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:12,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:12,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,531 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:12,531 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:12,531 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:12,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:12,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:12,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:12,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:12,531 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,531 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:12,531 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:12,531 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:12,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:12,532 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:12,531 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:12,532 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:12,532 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:12,532 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:12,532 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:12,632 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:12,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:12,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:12,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:12,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:12,638 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:12,638 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:12,638 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:12,638 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:12,638 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:12,638 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:12,639 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,639 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:12,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:12,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:12,639 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,639 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:12,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:12,640 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:12,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:12,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:12,640 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:12,640 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:12,640 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:12,640 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:12,641 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:12,641 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:12,641 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:12,641 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:12,641 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:12,641 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:12,641 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:12,641 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:12,641 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:12,641 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:12,641 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,641 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:12,641 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:12,642 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,642 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:12,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:12,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:12,642 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:12,642 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:12,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:12,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:12,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:12,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:12,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:12,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5126d625[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:12,644 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:12,644 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:12,644 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:12,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35892 deadline: 1681995372641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,644 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:12,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35842 deadline: 1681995372643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,645 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,645 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:12,646 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:12,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:12,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:12,646 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:12,646 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:12,646 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:12,646 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,646 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:12,647 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:12,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:12,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:12,647 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:12,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:12,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:12,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:12,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:12,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:12,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,649 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:12,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35900 deadline: 1681995372647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,649 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:12,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35848 deadline: 1681995372648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:12,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:12,651 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:12,651 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:12,651 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:12,651 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:12,651 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:12,651 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:12,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35886 deadline: 1681995372649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,652 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:12,652 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:12,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:12,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:12,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:12,652 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:12,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:12,652 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:12,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:12,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:12,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:12,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:12,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:12,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:12,653 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:12,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:12,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:12,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,655 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:12,655 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,655 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:12,655 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:12,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:12,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:12,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:12,655 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:12,656 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:12,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:12,655 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,655 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:12,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:12,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:12,656 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:12,656 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:12,656 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:12,656 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:12,656 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:12,739 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042050461b49fffc4f17abe51489b7f15404 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042050461b49fffc4f17abe51489b7f15404 2023-04-20 12:55:12,756 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:12,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:12,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:12,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:12,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:12,765 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:12,765 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:12,765 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:12,765 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:12,765 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:12,765 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:12,766 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,766 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:12,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:12,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:12,766 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,766 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:12,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:12,766 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:12,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:12,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:12,767 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:12,767 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:12,767 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:12,767 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:12,767 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:12,767 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:12,767 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:12,767 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:12,767 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:12,767 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:12,767 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:12,767 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:12,767 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:12,767 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:12,767 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,767 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:12,767 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:12,768 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,768 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:12,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:12,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:12,768 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:12,768 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:12,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:12,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:12,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:12,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:12,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:12,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39c79f8[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:12,769 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:12,769 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:12,770 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:12,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:12,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:12,770 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:12,770 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,770 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:12,770 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:12,770 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:12,771 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,771 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:12,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:12,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:12,771 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:12,771 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:12,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:12,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:12,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:12,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:12,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:12,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:12,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:12,772 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:12,772 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:12,772 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:12,772 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:12,772 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:12,773 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:12,773 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:12,773 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:12,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:12,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:12,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:12,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:12,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:12,774 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:12,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:12,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:12,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:12,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:12,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:12,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:12,774 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:12,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:12,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,776 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,776 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:12,776 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:12,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:12,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:12,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:12,776 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:12,776 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:12,776 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:12,776 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,776 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:12,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:12,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:12,776 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:12,776 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:12,776 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:12,776 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:12,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:12,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:12,876 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:12,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:12,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:12,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:12,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:12,880 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:12,880 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:12,880 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:12,880 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:12,881 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:12,881 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:12,881 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,881 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:12,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:12,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:12,882 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,882 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:12,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:12,882 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:12,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:12,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:12,882 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:12,882 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:12,882 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:12,882 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:12,882 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:12,882 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:12,883 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:12,883 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:12,883 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:12,883 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:12,883 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:12,883 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:12,883 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:12,883 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:12,883 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,883 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:12,883 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:12,884 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:12,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:12,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:12,884 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:12,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:12,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:12,885 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:12,885 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:12,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:12,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:12,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3508a0b5[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:12,886 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:12,886 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:12,886 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:12,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:12,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:12,886 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:12,886 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:12,886 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,886 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:12,886 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:12,887 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,887 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:12,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:12,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:12,887 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:12,888 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:12,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:12,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:12,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:12,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:12,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:12,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:12,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:12,889 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:12,889 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:12,889 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:12,889 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:12,889 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:12,890 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:12,890 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:12,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:12,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:12,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:12,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:12,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:12,891 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,891 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:12,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:12,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:12,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:12,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:12,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:12,892 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:12,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:12,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:12,895 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,895 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:12,895 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:12,895 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:12,895 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:12,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:12,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:12,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:12,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:12,895 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:12,895 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:12,895 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,895 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:12,895 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:12,895 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:12,895 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:12,895 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:12,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:12,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:12,948 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:12,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35892 deadline: 1681995372945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,949 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:12,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35842 deadline: 1681995372945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,955 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:12,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35900 deadline: 1681995372952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,955 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:12,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35848 deadline: 1681995372952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,957 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:12,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35886 deadline: 1681995372954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:12,995 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:12,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:12,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:12,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:12,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:12,999 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:12,999 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:12,999 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:12,999 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:12,999 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:12,999 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:13,000 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:13,000 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:13,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:13,000 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:13,000 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:13,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:13,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:13,001 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:13,001 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:13,001 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:13,001 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:13,001 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:13,001 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:13,001 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:13,001 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:13,001 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:13,001 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:13,001 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:13,001 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:13,001 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:13,001 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:13,001 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,001 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:13,001 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:13,002 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,002 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:13,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:13,003 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,003 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:13,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:13,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:13,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:13,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:13,004 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:13,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b74702a[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:13,004 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,004 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,004 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:13,004 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:13,004 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:13,005 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,005 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:13,005 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:13,005 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:13,005 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:13,005 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:13,005 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:13,005 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:13,005 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:13,005 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:13,005 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:13,006 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:13,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:13,006 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,006 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:13,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:13,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:13,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:13,008 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:13,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,009 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:13,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,010 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,010 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:13,010 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:13,010 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:13,010 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:13,010 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:13,010 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,010 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:13,010 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,010 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,010 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:13,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:13,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:13,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:13,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:13,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:13,018 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:13,018 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:13,118 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:13,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:13,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:13,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:13,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:13,121 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:13,121 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:13,121 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:13,121 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:13,121 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,121 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:13,122 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:13,122 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:13,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:13,122 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,122 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:13,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:13,122 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:13,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:13,122 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:13,123 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:13,123 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:13,123 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:13,123 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:13,123 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:13,123 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:13,123 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:13,123 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:13,123 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:13,123 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:13,123 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:13,123 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:13,123 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,123 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:13,123 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:13,124 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,124 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:13,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:13,124 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,124 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:13,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:13,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:13,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:13,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:13,126 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:13,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c84009f[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:13,126 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,127 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,127 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:13,127 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:13,127 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,127 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:13,127 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:13,128 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:13,128 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:13,128 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,128 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:13,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:13,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:13,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:13,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:13,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:13,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:13,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:13,130 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:13,130 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:13,130 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:13,130 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:13,130 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:13,131 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,131 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:13,131 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:13,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:13,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,131 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:13,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:13,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:13,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:13,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,132 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:13,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,133 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,133 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:13,133 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:13,133 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,133 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,133 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:13,133 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:13,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:13,133 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:13,133 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:13,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:13,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:13,133 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:13,133 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:13,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:13,135 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:13,135 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:13,143 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/A/eec2a36dffba485eb8ea3ae39f3bc619 2023-04-20 12:55:13,235 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:13,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:13,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:13,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:13,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:13,238 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:13,238 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:13,238 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:13,238 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:13,238 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,238 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:13,239 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,239 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:13,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:13,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:13,239 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,239 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:13,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:13,239 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:13,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:13,239 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:13,240 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:13,240 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:13,240 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:13,240 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:13,240 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:13,240 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:13,240 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:13,240 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:13,240 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:13,240 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:13,240 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:13,240 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:13,240 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:13,240 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,240 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:13,240 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:13,241 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,241 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:13,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:13,241 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,241 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:13,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:13,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:13,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:13,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:13,242 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:13,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4cb7ff4e[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:13,242 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,243 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,243 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,243 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:13,243 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:13,243 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:13,243 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:13,244 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,244 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:13,244 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:13,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:13,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:13,244 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:13,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:13,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:13,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:13,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:13,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:13,246 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:13,246 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:13,246 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:13,246 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:13,246 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:13,246 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,246 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:13,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:13,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,246 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:13,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,247 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:13,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:13,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:13,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:13,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,248 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:13,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,251 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,251 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:13,251 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:13,251 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:13,251 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:13,251 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:13,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:13,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:13,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:13,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:13,251 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,251 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,251 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:13,251 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,251 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:13,251 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:13,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:13,252 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:13,352 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:13,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:13,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:13,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:13,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:13,354 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:13,354 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:13,354 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:13,354 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:13,355 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,355 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:13,355 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:13,356 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:13,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:13,356 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,356 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:13,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:13,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:13,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:13,356 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:13,357 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:13,357 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:13,357 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:13,357 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:13,357 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:13,357 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:13,357 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:13,357 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:13,357 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:13,357 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:13,357 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:13,357 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:13,357 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:13,357 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,357 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:13,357 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:13,358 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:13,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:13,358 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:13,358 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,358 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:13,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:13,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:13,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:13,359 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:13,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c4ae365[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:13,359 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,360 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,360 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:13,360 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:13,360 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,360 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:13,360 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:13,361 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,361 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:13,361 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:13,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:13,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:13,361 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:13,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:13,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:13,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:13,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:13,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:13,362 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:13,362 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:13,363 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:13,363 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:13,363 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:13,363 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,363 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:13,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:13,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,363 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:13,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:13,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:13,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:13,364 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:13,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,365 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:13,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,366 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,366 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:13,366 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:13,366 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:13,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:13,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:13,367 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,367 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:13,367 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:13,367 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:13,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:13,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:13,367 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,367 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:13,367 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:13,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:13,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:13,367 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,455 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:13,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35842 deadline: 1681995373453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,455 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:13,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35892 deadline: 1681995373453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,467 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:13,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:13,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:13,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:13,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:13,470 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:13,471 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:13,471 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:13,471 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:13,471 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,471 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:13,472 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,472 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:13,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:13,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:13,472 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:13,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35848 deadline: 1681995373468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,472 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,472 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:13,472 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:13,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35886 deadline: 1681995373468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,473 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:13,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35900 deadline: 1681995373468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:13,474 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:13,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:13,475 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:13,475 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:13,475 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:13,475 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:13,475 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:13,475 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:13,479 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:13,479 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:13,479 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:13,479 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:13,479 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:13,479 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:13,479 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:13,479 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:13,479 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,479 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:13,479 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:13,482 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,482 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,482 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:13,483 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:13,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:13,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:13,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:13,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:13,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:13,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54d731e6[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:13,484 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:13,484 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,494 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,494 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,494 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:13,494 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:13,494 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:13,494 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:13,498 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,498 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:13,498 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:13,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:13,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:13,498 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:13,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:13,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:13,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:13,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:13,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:13,500 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:13,500 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:13,500 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:13,500 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:13,500 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:13,501 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,501 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:13,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:13,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,501 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:13,502 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:13,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:13,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:13,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:13,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,503 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:13,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,504 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,504 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:13,504 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:13,504 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,504 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:13,504 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,504 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:13,504 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:13,504 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:13,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:13,504 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,504 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:13,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:13,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:13,504 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:13,504 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:13,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:13,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:13,559 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/B/4b6c335ee8134907b2f4484d0eb9f3a1 2023-04-20 12:55:13,604 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:13,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:13,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:13,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:13,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:13,607 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:13,607 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:13,607 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:13,607 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:13,608 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,608 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:13,608 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:13,608 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:13,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:13,608 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,609 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:13,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:13,609 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:13,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:13,609 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:13,609 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:13,609 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:13,609 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:13,609 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:13,609 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:13,609 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:13,609 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:13,609 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:13,609 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:13,609 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:13,610 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:13,610 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:13,610 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:13,610 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,610 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:13,610 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:13,610 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:13,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:13,610 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:13,611 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,611 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:13,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:13,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:13,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:13,612 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:13,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58db431b[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:13,612 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,612 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,613 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:13,613 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:13,613 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:13,613 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,613 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:13,613 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,613 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:13,613 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:13,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:13,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:13,613 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:13,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:13,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:13,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:13,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:13,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:13,616 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:13,616 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:13,616 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:13,616 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:13,616 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:13,617 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,617 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:13,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:13,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,617 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:13,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:13,618 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:13,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:13,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:13,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,619 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:13,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,620 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,620 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:13,620 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:13,620 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,620 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,620 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:13,620 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:13,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:13,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:13,620 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:13,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:13,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:13,620 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,620 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,621 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:13,621 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:13,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:13,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:13,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:13,721 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:13,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:13,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:13,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:13,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:13,724 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:13,724 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:13,724 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:13,724 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:13,724 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,724 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:13,726 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:13,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:13,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:13,726 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:13,726 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,726 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:13,727 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:13,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:13,727 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:13,727 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:13,727 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:13,727 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:13,727 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:13,727 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:13,727 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:13,727 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:13,727 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:13,727 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:13,727 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:13,727 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:13,728 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:13,728 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:13,728 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,728 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:13,728 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:13,728 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,728 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:13,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:13,729 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,729 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:13,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:13,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:13,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:13,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:13,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2be59ac1[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:13,730 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:13,730 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,730 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,730 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:13,731 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:13,730 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,731 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:13,731 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:13,732 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:13,732 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:13,732 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:13,733 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:13,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:13,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:13,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:13,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:13,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:13,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:13,737 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:13,737 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:13,737 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:13,737 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:13,737 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:13,738 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,738 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,739 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:13,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:13,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:13,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:13,741 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:13,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:13,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:13,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,742 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:13,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,745 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:13,745 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:13,746 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,746 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:13,746 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:13,746 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,746 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,746 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:13,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:13,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:13,746 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:13,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:13,746 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,746 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:13,746 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:13,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:13,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:13,846 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:13,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:13,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:13,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:13,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:13,849 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:13,849 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:13,849 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:13,849 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:13,849 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,849 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:13,850 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,850 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:13,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:13,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:13,850 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,850 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:13,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:13,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:13,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:13,851 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:13,851 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:13,851 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:13,851 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:13,851 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:13,851 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:13,851 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:13,851 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:13,851 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:13,851 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:13,851 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:13,851 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:13,851 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:13,851 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:13,851 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,851 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:13,851 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:13,852 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,852 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:13,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:13,852 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,852 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:13,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:13,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:13,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:13,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:13,853 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:13,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40df1513[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:13,853 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,854 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,854 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:13,854 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:13,854 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,854 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:13,854 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:13,855 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,855 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:13,855 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:13,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:13,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:13,855 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:13,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:13,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:13,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:13,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:13,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:13,857 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:13,857 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:13,857 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:13,857 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:13,857 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:13,857 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,858 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:13,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:13,858 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:13,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:13,858 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:13,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:13,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:13,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,859 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:13,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,860 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,860 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:13,860 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:13,860 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:13,860 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:13,860 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:13,860 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:13,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:13,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:13,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:13,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:13,860 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,861 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,861 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,861 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:13,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:13,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:13,861 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:13,961 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:13,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:13,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:13,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:13,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:13,963 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:13,963 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:13,963 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:13,963 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:13,964 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,964 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:13,964 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:13,964 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:13,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:13,965 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,965 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:13,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:13,965 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:13,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:13,965 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:13,965 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:13,965 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:13,965 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:13,965 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:13,966 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:13,966 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:13,966 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:13,966 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:13,966 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:13,966 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:13,966 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:13,966 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:13,966 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:13,966 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,966 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:13,966 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:13,967 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,967 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:13,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:13,967 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,967 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:13,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:13,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:13,967 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/C/514cca0401b24ecb87cbb3c0c4a5f63a 2023-04-20 12:55:13,971 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/A/eec2a36dffba485eb8ea3ae39f3bc619 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/eec2a36dffba485eb8ea3ae39f3bc619 2023-04-20 12:55:13,973 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-20 12:55:13,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,974 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/eec2a36dffba485eb8ea3ae39f3bc619, entries=150, sequenceid=16, filesize=25.1 K 2023-04-20 12:55:13,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,975 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/B/4b6c335ee8134907b2f4484d0eb9f3a1 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/4b6c335ee8134907b2f4484d0eb9f3a1 2023-04-20 12:55:13,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:13,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:13,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d6a54ef[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:13,976 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:13,976 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,979 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/4b6c335ee8134907b2f4484d0eb9f3a1, entries=150, sequenceid=16, filesize=11.6 K 2023-04-20 12:55:13,980 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/C/514cca0401b24ecb87cbb3c0c4a5f63a as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/514cca0401b24ecb87cbb3c0c4a5f63a 2023-04-20 12:55:13,982 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,982 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:13,982 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:13,982 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:13,983 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,983 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:13,986 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:13,986 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,986 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:13,986 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:13,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:13,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:13,987 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/514cca0401b24ecb87cbb3c0c4a5f63a, entries=150, sequenceid=16, filesize=11.6 K 2023-04-20 12:55:13,988 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for aeecfead619437b2701aee4eba741c09 in 1682ms, sequenceid=16, compaction requested=false 2023-04-20 12:55:13,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:13,988 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for aeecfead619437b2701aee4eba741c09: 2023-04-20 12:55:13,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:13,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:13,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:13,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:13,990 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:13,990 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:13,990 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:13,990 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:13,990 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:13,994 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:13,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:13,994 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,994 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:13,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:13,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:13,995 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:13,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:13,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,996 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:13,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:13,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,998 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,999 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:13,999 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:13,999 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:13,999 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,999 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:13,999 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:13,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:13,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:13,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:13,999 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:14,000 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:14,000 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:14,000 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:14,000 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:14,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:14,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:14,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:14,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:14,100 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:14,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:14,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:14,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:14,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:14,104 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:14,104 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:14,104 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:14,104 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:14,105 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:14,105 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:14,105 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:14,105 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:14,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:14,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:14,106 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:14,106 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:14,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:14,106 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:14,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:14,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:14,106 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:14,106 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:14,106 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:14,107 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:14,107 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:14,107 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:14,107 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:14,107 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:14,107 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:14,107 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:14,107 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing aeecfead619437b2701aee4eba741c09 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-20 12:55:14,107 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=A 2023-04-20 12:55:14,107 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:14,107 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=B 2023-04-20 12:55:14,107 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:14,107 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=C 2023-04-20 12:55:14,107 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:14,129 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304205c7e5ad6a03847b6900690881fe31e9d to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304205c7e5ad6a03847b6900690881fe31e9d 2023-04-20 12:55:14,137 WARN [DataStreamer for file /user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/A/99a9cb3d3752450fbd4ac7b9289c9c79] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-04-20 12:55:14,137 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=36, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/A/99a9cb3d3752450fbd4ac7b9289c9c79 2023-04-20 12:55:14,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on aeecfead619437b2701aee4eba741c09 2023-04-20 12:55:14,461 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:14,484 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:14,485 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:14,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35842 deadline: 1681995374478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:14,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35892 deadline: 1681995374478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:14,487 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:14,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35886 deadline: 1681995374481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:14,487 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:14,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35848 deadline: 1681995374481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:14,489 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:14,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35900 deadline: 1681995374485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:14,567 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/B/42d5f85f8ac948e6bd7eef8c7142faa8 2023-04-20 12:55:14,590 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:14,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35842 deadline: 1681995374586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:14,591 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:14,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35892 deadline: 1681995374587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:14,593 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:14,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35848 deadline: 1681995374588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:14,593 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:14,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35900 deadline: 1681995374590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:14,798 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:14,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35842 deadline: 1681995374793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:14,799 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:14,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35892 deadline: 1681995374794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:14,799 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:14,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35848 deadline: 1681995374794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:14,799 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:14,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35900 deadline: 1681995374795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,014 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/C/4ca7b4e2c47c4fbdaefd77487f4884c0 2023-04-20 12:55:15,020 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/A/99a9cb3d3752450fbd4ac7b9289c9c79 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/99a9cb3d3752450fbd4ac7b9289c9c79 2023-04-20 12:55:15,025 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/99a9cb3d3752450fbd4ac7b9289c9c79, entries=150, sequenceid=36, filesize=25.1 K 2023-04-20 12:55:15,025 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/B/42d5f85f8ac948e6bd7eef8c7142faa8 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/42d5f85f8ac948e6bd7eef8c7142faa8 2023-04-20 12:55:15,032 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/42d5f85f8ac948e6bd7eef8c7142faa8, entries=150, sequenceid=36, filesize=11.6 K 2023-04-20 12:55:15,034 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/C/4ca7b4e2c47c4fbdaefd77487f4884c0 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/4ca7b4e2c47c4fbdaefd77487f4884c0 2023-04-20 12:55:15,040 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/4ca7b4e2c47c4fbdaefd77487f4884c0, entries=150, sequenceid=36, filesize=11.6 K 2023-04-20 12:55:15,041 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for aeecfead619437b2701aee4eba741c09 in 934ms, sequenceid=36, compaction requested=false 2023-04-20 12:55:15,041 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for aeecfead619437b2701aee4eba741c09: 2023-04-20 12:55:15,041 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:15,041 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:15,041 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:15,041 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,042 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:15,042 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:15,043 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,043 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:15,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:15,044 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,044 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:15,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:15,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:15,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:15,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:15,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:15,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fab09c2[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:15,047 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:15,047 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,047 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,048 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:15,048 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:15,048 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:15,048 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,048 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:15,049 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,049 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:15,049 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:15,049 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:15,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:15,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:15,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:15,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:15,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:15,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:15,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:15,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:15,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:15,051 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:15,051 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:15,051 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:15,051 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:15,051 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:15,053 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,053 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:15,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,053 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:15,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:15,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:15,054 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:15,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:15,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:15,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:15,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:15,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,058 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:15,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:15,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,060 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,060 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:15,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:15,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:15,060 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:15,060 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:15,060 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:15,060 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,060 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:15,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:15,060 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:15,060 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,060 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:15,060 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:15,060 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:15,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:15,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on aeecfead619437b2701aee4eba741c09 2023-04-20 12:55:15,106 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing aeecfead619437b2701aee4eba741c09 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-20 12:55:15,107 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=A 2023-04-20 12:55:15,107 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:15,107 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=B 2023-04-20 12:55:15,107 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:15,107 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=C 2023-04-20 12:55:15,107 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:15,130 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304205af21a6a19684d1694680bd9793cfa55 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304205af21a6a19684d1694680bd9793cfa55 2023-04-20 12:55:15,136 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:15,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35892 deadline: 1681995375128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,141 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:15,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35848 deadline: 1681995375133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,144 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:15,144 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:15,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35842 deadline: 1681995375134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35900 deadline: 1681995375136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,160 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:15,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:15,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:15,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:15,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:15,165 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:15,165 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:15,165 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:15,165 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:15,166 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,166 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:15,167 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:15,167 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:15,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:15,168 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,168 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:15,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:15,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:15,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:15,169 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:15,169 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:15,169 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:15,169 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:15,169 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:15,169 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:15,170 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:15,170 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:15,170 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:15,170 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:15,170 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:15,170 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:15,170 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:15,170 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:15,170 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,170 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:15,170 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:15,171 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,172 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:15,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:15,172 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,172 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:15,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:15,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:15,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:15,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:15,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:15,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@281fad64[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:15,175 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:15,175 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,175 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,175 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:15,175 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:15,175 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,175 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:15,175 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:15,176 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,176 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:15,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:15,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:15,176 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:15,176 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:15,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:15,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:15,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:15,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:15,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:15,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:15,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:15,178 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:15,178 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:15,178 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:15,178 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:15,178 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:15,180 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,180 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:15,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,180 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:15,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:15,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:15,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:15,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:15,181 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:15,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:15,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:15,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,182 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:15,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:15,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,183 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,183 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:15,183 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:15,183 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:15,183 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:15,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:15,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:15,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:15,183 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:15,184 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,184 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:15,184 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:15,184 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:15,184 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:15,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:15,184 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,241 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:15,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35892 deadline: 1681995375237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,245 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:15,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35848 deadline: 1681995375242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,248 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:15,248 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:15,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35900 deadline: 1681995375245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35842 deadline: 1681995375245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,284 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:15,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:15,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:15,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:15,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:15,291 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:15,291 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:15,291 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:15,291 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:15,291 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,291 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:15,292 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:15,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:15,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:15,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:15,292 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,293 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,293 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,293 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:15,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:15,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:15,293 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:15,294 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:15,294 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:15,294 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:15,294 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:15,294 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:15,298 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:15,298 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:15,298 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:15,298 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:15,298 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:15,298 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:15,298 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:15,298 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:15,298 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,298 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:15,298 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:15,307 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,307 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:15,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:15,307 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,307 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:15,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:15,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:15,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:15,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:15,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:15,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@105042f9[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:15,309 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:15,309 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,311 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,311 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:15,311 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,311 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:15,311 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:15,311 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:15,312 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:15,312 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:15,312 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:15,313 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:15,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:15,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:15,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:15,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:15,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:15,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:15,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:15,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:15,315 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:15,315 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:15,315 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:15,315 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:15,316 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:15,316 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,317 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:15,317 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:15,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:15,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:15,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:15,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:15,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,318 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:15,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:15,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:15,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,321 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:15,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:15,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,323 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,323 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:15,323 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:15,324 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:15,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:15,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:15,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:15,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:15,323 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:15,324 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:15,324 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,324 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,324 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,324 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:15,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:15,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:15,324 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,324 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:15,424 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:15,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:15,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:15,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:15,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:15,432 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:15,432 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:15,432 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:15,432 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:15,433 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,433 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:15,434 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,434 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:15,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:15,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:15,434 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,434 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:15,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:15,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:15,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:15,435 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:15,435 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:15,435 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:15,435 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:15,435 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:15,435 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:15,439 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:15,439 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:15,439 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:15,439 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:15,439 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:15,439 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:15,439 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:15,439 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:15,439 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,439 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:15,439 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:15,440 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,440 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:15,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:15,441 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,441 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:15,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:15,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:15,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:15,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:15,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:15,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fe2b130[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:15,442 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:15,442 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,442 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,442 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:15,442 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:15,442 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,443 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:15,442 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:15,444 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,444 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:15,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:15,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:15,444 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:15,444 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:15,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:15,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:15,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:15,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:15,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:15,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,448 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:15,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:15,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35892 deadline: 1681995375443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,448 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:15,448 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:15,448 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:15,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:15,448 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:15,448 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:15,448 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:15,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35848 deadline: 1681995375446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,455 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:15,455 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:15,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35842 deadline: 1681995375450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35900 deadline: 1681995375451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,457 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:15,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:15,457 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,457 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:15,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,457 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:15,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:15,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:15,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:15,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:15,458 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:15,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:15,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:15,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,461 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,461 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:15,461 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:15,461 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:15,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:15,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:15,461 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:15,461 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,461 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,461 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,461 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,461 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:15,461 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:15,461 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:15,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:15,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:15,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:15,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:15,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:15,544 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/A/77a1caf6b7ef4e9a8b60dc22adb122d3 2023-04-20 12:55:15,562 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:15,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:15,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:15,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:15,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:15,564 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:15,564 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:15,564 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:15,564 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:15,565 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,565 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:15,565 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:15,565 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:15,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:15,566 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,566 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:15,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:15,566 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:15,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:15,566 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:15,566 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:15,566 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:15,567 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:15,567 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:15,567 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:15,568 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:15,568 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:15,568 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:15,568 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:15,568 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:15,568 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:15,568 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:15,568 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:15,568 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,568 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:15,569 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:15,569 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,569 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:15,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:15,569 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,570 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:15,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:15,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:15,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:15,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:15,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:15,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18705017[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:15,578 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:15,578 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,582 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,582 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:15,582 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:15,582 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:15,582 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,582 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:15,583 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,583 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:15,583 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:15,583 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:15,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:15,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:15,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:15,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:15,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:15,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:15,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:15,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:15,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:15,586 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:15,587 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:15,587 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:15,587 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:15,587 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:15,587 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,587 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:15,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:15,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,587 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:15,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:15,588 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:15,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:15,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:15,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:15,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:15,589 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:15,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:15,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,590 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:15,590 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:15,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:15,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:15,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:15,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:15,594 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,594 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:15,594 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:15,594 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,594 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:15,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:15,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:15,594 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,594 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:15,594 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,594 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:15,594 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:15,694 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:15,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:15,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:15,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:15,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:15,697 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:15,697 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:15,697 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:15,697 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:15,698 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,698 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:15,698 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,698 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:15,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:15,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:15,698 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,698 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:15,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:15,699 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:15,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:15,699 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:15,699 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:15,699 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:15,699 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:15,699 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:15,700 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:15,700 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:15,700 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:15,700 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:15,700 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:15,700 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:15,700 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:15,700 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:15,700 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:15,700 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,700 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:15,700 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:15,701 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,701 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:15,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:15,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:15,701 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,701 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:15,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:15,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:15,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:15,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:15,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d51baf0[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:15,702 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:15,702 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,703 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,703 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,703 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:15,703 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:15,703 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:15,703 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:15,704 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:15,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:15,704 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:15,704 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:15,704 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:15,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:15,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:15,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:15,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:15,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:15,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:15,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:15,705 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:15,705 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:15,706 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:15,706 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:15,706 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:15,706 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,706 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:15,706 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:15,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:15,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:15,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:15,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:15,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,707 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:15,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:15,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:15,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,708 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:15,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:15,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,709 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:15,709 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:15,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:15,709 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:15,709 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:15,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:15,709 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:15,709 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:15,710 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:15,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:15,709 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:15,709 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:15,710 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,710 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,710 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,710 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:15,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:15,754 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:15,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35892 deadline: 1681995375751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,754 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:15,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35848 deadline: 1681995375752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,762 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:15,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35900 deadline: 1681995375757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,763 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:15,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35842 deadline: 1681995375758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,810 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:15,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:15,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:15,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:15,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:15,813 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:15,813 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:15,813 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:15,813 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:15,813 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,813 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:15,814 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:15,814 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:15,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:15,814 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,814 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:15,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:15,814 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:15,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:15,815 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:15,815 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:15,815 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:15,815 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:15,815 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:15,815 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:15,817 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:15,817 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:15,818 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:15,818 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:15,818 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:15,818 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:15,818 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:15,819 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:15,819 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,819 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:15,819 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:15,827 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,827 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:15,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:15,828 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,828 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:15,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:15,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:15,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:15,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:15,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:15,829 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:15,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f046e29[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:15,829 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,832 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,832 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:15,832 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,832 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:15,832 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:15,832 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:15,833 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,833 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:15,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:15,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:15,833 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:15,833 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:15,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:15,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:15,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:15,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:15,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:15,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:15,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:15,837 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:15,837 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:15,837 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:15,837 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:15,837 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:15,844 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,844 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:15,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:15,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,845 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:15,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,845 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:15,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:15,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:15,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:15,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:15,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:15,846 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:15,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:15,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,848 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,848 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:15,848 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:15,848 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,848 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,848 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:15,849 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:15,848 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:15,849 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:15,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:15,849 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:15,849 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:15,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:15,849 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:15,849 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:15,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:15,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:15,949 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:15,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:15,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:15,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:15,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:15,952 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:15,952 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:15,952 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:15,952 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:15,952 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/B/373728976e774018a06afe9223645f63 2023-04-20 12:55:15,953 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,953 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:15,953 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:15,953 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:15,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:15,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:15,954 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,954 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:15,954 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:15,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:15,954 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:15,954 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:15,954 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:15,954 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:15,954 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:15,954 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:15,955 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:15,955 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:15,955 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:15,955 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:15,955 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:15,955 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:15,955 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:15,955 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:15,955 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,955 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:15,955 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:15,956 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,956 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:15,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:15,957 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:15,957 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:15,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:15,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:15,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:15,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:15,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25952fc8[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:15,958 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:15,958 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,959 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,959 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,959 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:15,959 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:15,959 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:15,959 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:15,960 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,960 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:15,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:15,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:15,960 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:15,960 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:15,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:15,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:15,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:15,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:15,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:15,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:15,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:15,962 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:15,962 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:15,962 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:15,962 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:15,962 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:15,963 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,963 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:15,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:15,963 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:15,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:15,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:15,963 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:15,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:15,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:15,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:15,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,965 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:15,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:15,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,966 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,966 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:15,966 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:15,966 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:15,966 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,966 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:15,966 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:15,966 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:15,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:15,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:15,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:15,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:15,966 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:15,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:15,966 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:15,966 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:15,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:15,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:15,966 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:16,067 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:16,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:16,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:16,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:16,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:16,070 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:16,070 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:16,070 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:16,070 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:16,070 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:16,070 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:16,071 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,071 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:16,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:16,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:16,071 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,071 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:16,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:16,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:16,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:16,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:16,071 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:16,072 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:16,072 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:16,072 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:16,072 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:16,072 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:16,072 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:16,072 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:16,072 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:16,072 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:16,072 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:16,072 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:16,072 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:16,072 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,072 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:16,072 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:16,073 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,073 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:16,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:16,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:16,073 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:16,073 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:16,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:16,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:16,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:16,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:16,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:16,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f7f3385[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:16,074 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:16,074 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:16,076 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:16,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:16,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:16,076 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:16,076 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:16,076 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:16,076 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,076 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:16,077 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:16,077 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,077 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:16,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:16,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:16,077 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:16,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:16,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:16,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:16,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:16,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:16,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:16,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:16,079 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:16,079 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:16,079 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:16,079 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:16,079 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:16,080 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:16,080 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:16,080 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:16,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:16,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:16,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:16,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:16,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:16,080 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:16,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:16,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:16,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:16,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:16,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:16,081 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:16,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:16,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:16,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,082 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,082 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:16,082 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:16,082 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:16,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:16,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:16,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:16,082 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:16,082 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:16,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:16,082 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,083 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:16,083 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:16,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:16,083 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:16,083 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:16,083 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:16,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:16,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:16,183 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:16,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:16,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:16,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:16,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:16,185 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:16,186 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:16,186 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:16,186 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:16,186 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:16,186 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:16,187 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:16,187 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:16,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:16,187 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,187 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:16,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:16,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:16,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:16,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:16,187 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:16,188 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:16,188 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:16,188 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:16,188 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:16,188 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:16,188 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:16,188 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:16,188 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:16,188 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:16,188 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:16,188 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:16,188 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:16,188 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:16,188 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,188 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:16,188 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:16,189 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:16,189 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:16,189 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:16,189 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:16,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:16,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:16,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:16,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:16,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:16,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:16,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fb988a0[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:16,191 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:16,191 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:16,192 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,192 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:16,192 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:16,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:16,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:16,192 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:16,192 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:16,192 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:16,193 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,193 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:16,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:16,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:16,193 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:16,193 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:16,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:16,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:16,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:16,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:16,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:16,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:16,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:16,195 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:16,195 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:16,195 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:16,195 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:16,195 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:16,196 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:16,196 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:16,196 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:16,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:16,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:16,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:16,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:16,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:16,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:16,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:16,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:16,197 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:16,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:16,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:16,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:16,198 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:16,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:16,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,200 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,200 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:16,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:16,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:16,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:16,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:16,200 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:16,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:16,200 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:16,200 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:16,200 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:16,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:16,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:16,200 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:16,200 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:16,201 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,201 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:16,201 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:16,201 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:16,257 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:16,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35848 deadline: 1681995376256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,260 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:16,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35892 deadline: 1681995376257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,268 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:16,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35900 deadline: 1681995376266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,269 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:16,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35842 deadline: 1681995376267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,301 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:16,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:16,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:16,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:16,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:16,303 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:16,303 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:16,303 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:16,303 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:16,304 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:16,304 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:16,304 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:16,304 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:16,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:16,305 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,305 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:16,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:16,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:16,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:16,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:16,305 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:16,305 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:16,305 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:16,306 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:16,306 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:16,306 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:16,306 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:16,306 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:16,306 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:16,306 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:16,306 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:16,306 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:16,306 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:16,306 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:16,306 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,306 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:16,306 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:16,307 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,307 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:16,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:16,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:16,307 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:16,307 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:16,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:16,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:16,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:16,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:16,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:16,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@330d458[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:16,308 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:16,308 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:16,309 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:16,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:16,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:16,309 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:16,309 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,309 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:16,309 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:16,309 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:16,310 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,310 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:16,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:16,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:16,310 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:16,310 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:16,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:16,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:16,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:16,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:16,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:16,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:16,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:16,311 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:16,311 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:16,311 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:16,311 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:16,312 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:16,312 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:16,312 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:16,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:16,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:16,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:16,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:16,312 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:16,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:16,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:16,313 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:16,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:16,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:16,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:16,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:16,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:16,314 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:16,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:16,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,315 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,315 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:16,315 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:16,315 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:16,315 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:16,315 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:16,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:16,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:16,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:16,315 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:16,315 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:16,315 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:16,315 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:16,315 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:16,315 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:16,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:16,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:16,315 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:16,363 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/C/a9dd16fe8dcb45aa8743a9c382fd94d9 2023-04-20 12:55:16,366 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/A/77a1caf6b7ef4e9a8b60dc22adb122d3 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/77a1caf6b7ef4e9a8b60dc22adb122d3 2023-04-20 12:55:16,374 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/77a1caf6b7ef4e9a8b60dc22adb122d3, entries=150, sequenceid=50, filesize=25.1 K 2023-04-20 12:55:16,374 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/B/373728976e774018a06afe9223645f63 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/373728976e774018a06afe9223645f63 2023-04-20 12:55:16,378 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/373728976e774018a06afe9223645f63, entries=150, sequenceid=50, filesize=11.6 K 2023-04-20 12:55:16,379 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/C/a9dd16fe8dcb45aa8743a9c382fd94d9 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/a9dd16fe8dcb45aa8743a9c382fd94d9 2023-04-20 12:55:16,391 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/a9dd16fe8dcb45aa8743a9c382fd94d9, entries=150, sequenceid=50, filesize=11.6 K 2023-04-20 12:55:16,392 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for aeecfead619437b2701aee4eba741c09 in 1286ms, sequenceid=50, compaction requested=true 2023-04-20 12:55:16,392 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for aeecfead619437b2701aee4eba741c09: 2023-04-20 12:55:16,393 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:55:16,393 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:55:16,393 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:55:16,393 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:55:16,393 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:55:16,395 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:55:16,395 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): aeecfead619437b2701aee4eba741c09/B is initiating minor compaction (all files) 2023-04-20 12:55:16,395 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of aeecfead619437b2701aee4eba741c09/B in TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:16,396 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/4b6c335ee8134907b2f4484d0eb9f3a1, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/42d5f85f8ac948e6bd7eef8c7142faa8, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/373728976e774018a06afe9223645f63] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp, totalSize=34.8 K 2023-04-20 12:55:16,396 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77238 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:55:16,396 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): aeecfead619437b2701aee4eba741c09/A is initiating minor compaction (all files) 2023-04-20 12:55:16,396 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of aeecfead619437b2701aee4eba741c09/A in TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:16,396 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 4b6c335ee8134907b2f4484d0eb9f3a1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1681995312275 2023-04-20 12:55:16,396 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/eec2a36dffba485eb8ea3ae39f3bc619, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/99a9cb3d3752450fbd4ac7b9289c9c79, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/77a1caf6b7ef4e9a8b60dc22adb122d3] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp, totalSize=75.4 K 2023-04-20 12:55:16,397 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 42d5f85f8ac948e6bd7eef8c7142faa8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1681995312327 2023-04-20 12:55:16,397 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting eec2a36dffba485eb8ea3ae39f3bc619, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1681995312275 2023-04-20 12:55:16,397 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 373728976e774018a06afe9223645f63, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681995314478 2023-04-20 12:55:16,398 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 99a9cb3d3752450fbd4ac7b9289c9c79, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1681995312327 2023-04-20 12:55:16,398 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 77a1caf6b7ef4e9a8b60dc22adb122d3, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681995314478 2023-04-20 12:55:16,405 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): aeecfead619437b2701aee4eba741c09#B#compaction#310 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:55:16,413 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): aeecfead619437b2701aee4eba741c09#A#compaction#311 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:55:16,416 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:16,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:16,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:16,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:16,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:16,426 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:16,426 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:16,426 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:16,426 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:16,426 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:16,427 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:16,428 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,429 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:16,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:16,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:16,429 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,429 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:16,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:16,429 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:16,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:16,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:16,429 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:16,430 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:16,430 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:16,430 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:16,430 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:16,430 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:16,430 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:16,430 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:16,430 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:16,430 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:16,430 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing aeecfead619437b2701aee4eba741c09 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-20 12:55:16,431 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=A 2023-04-20 12:55:16,431 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:16,431 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=B 2023-04-20 12:55:16,431 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:16,431 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=C 2023-04-20 12:55:16,431 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:16,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on aeecfead619437b2701aee4eba741c09 2023-04-20 12:55:16,495 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:16,566 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:16,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35886 deadline: 1681995376563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,670 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:16,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35886 deadline: 1681995376668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,816 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/B/efc5aabd716e4153bd07fc63309dc66e as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/efc5aabd716e4153bd07fc63309dc66e 2023-04-20 12:55:16,820 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in aeecfead619437b2701aee4eba741c09/B of aeecfead619437b2701aee4eba741c09 into efc5aabd716e4153bd07fc63309dc66e(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:55:16,820 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for aeecfead619437b2701aee4eba741c09: 2023-04-20 12:55:16,820 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09., storeName=aeecfead619437b2701aee4eba741c09/B, priority=13, startTime=1681995316393; duration=0sec 2023-04-20 12:55:16,820 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:55:16,820 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:55:16,821 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:55:16,821 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): aeecfead619437b2701aee4eba741c09/C is initiating minor compaction (all files) 2023-04-20 12:55:16,821 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of aeecfead619437b2701aee4eba741c09/C in TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:16,821 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/514cca0401b24ecb87cbb3c0c4a5f63a, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/4ca7b4e2c47c4fbdaefd77487f4884c0, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/a9dd16fe8dcb45aa8743a9c382fd94d9] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp, totalSize=34.8 K 2023-04-20 12:55:16,822 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 514cca0401b24ecb87cbb3c0c4a5f63a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1681995312275 2023-04-20 12:55:16,822 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 4ca7b4e2c47c4fbdaefd77487f4884c0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1681995312327 2023-04-20 12:55:16,822 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting a9dd16fe8dcb45aa8743a9c382fd94d9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681995314478 2023-04-20 12:55:16,826 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): aeecfead619437b2701aee4eba741c09#C#compaction#313 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:55:16,854 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420d70aa6429eba4c2db5494818dcdd63a0 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420d70aa6429eba4c2db5494818dcdd63a0 2023-04-20 12:55:16,860 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=69, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/A/633e06da32f34a27bd5efbbdd68f5b39 2023-04-20 12:55:16,874 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/B/c6313b421da44e3ca49f62f6f4e7e8b6 2023-04-20 12:55:16,877 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:16,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35886 deadline: 1681995376872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,888 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/C/d7c0f9df03e4416c8f865b607caa0f45 2023-04-20 12:55:16,899 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/A/633e06da32f34a27bd5efbbdd68f5b39 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/633e06da32f34a27bd5efbbdd68f5b39 2023-04-20 12:55:16,904 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/633e06da32f34a27bd5efbbdd68f5b39, entries=150, sequenceid=69, filesize=25.1 K 2023-04-20 12:55:16,906 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/B/c6313b421da44e3ca49f62f6f4e7e8b6 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/c6313b421da44e3ca49f62f6f4e7e8b6 2023-04-20 12:55:16,911 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/c6313b421da44e3ca49f62f6f4e7e8b6, entries=150, sequenceid=69, filesize=11.6 K 2023-04-20 12:55:16,912 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/C/d7c0f9df03e4416c8f865b607caa0f45 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/d7c0f9df03e4416c8f865b607caa0f45 2023-04-20 12:55:16,915 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/d7c0f9df03e4416c8f865b607caa0f45, entries=150, sequenceid=69, filesize=11.6 K 2023-04-20 12:55:16,916 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for aeecfead619437b2701aee4eba741c09 in 486ms, sequenceid=69, compaction requested=false 2023-04-20 12:55:16,916 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for aeecfead619437b2701aee4eba741c09: 2023-04-20 12:55:16,916 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:16,916 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:16,916 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:16,916 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,916 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:16,916 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:16,917 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,917 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:16,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:16,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:16,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:16,917 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:16,917 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:16,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:16,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:16,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:16,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:16,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f574099[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:16,919 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:16,919 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:16,920 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:16,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:16,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:16,920 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:16,920 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:16,920 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:16,920 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,920 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:16,920 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:16,920 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,920 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:16,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:16,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:16,921 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:16,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:16,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:16,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:16,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:16,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:16,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:16,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:16,922 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:16,922 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:16,922 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:16,922 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:16,922 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:16,923 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:16,923 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:16,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:16,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:16,923 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:16,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:16,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:16,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:16,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:16,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:16,924 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:16,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:16,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:16,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:16,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:16,924 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:16,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:16,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,926 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,926 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:16,926 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:16,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:16,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:16,926 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:16,926 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:16,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:16,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:16,926 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:16,926 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:16,926 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:16,926 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:16,926 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:16,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:16,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:16,926 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:16,926 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:16,926 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,026 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:17,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:17,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:17,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:17,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:17,029 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:17,029 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:17,029 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:17,029 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:17,029 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,029 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:17,030 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:17,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:17,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:17,030 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:17,030 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,030 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:17,031 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:17,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:17,031 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:17,031 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:17,031 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:17,031 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:17,031 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:17,031 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:17,031 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:17,031 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:17,031 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:17,031 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:17,031 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing aeecfead619437b2701aee4eba741c09 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-20 12:55:17,032 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=A 2023-04-20 12:55:17,032 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:17,032 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=B 2023-04-20 12:55:17,032 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:17,032 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=C 2023-04-20 12:55:17,032 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:17,047 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042042592e5513594dad90ba618c383ddd8d to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042042592e5513594dad90ba618c383ddd8d 2023-04-20 12:55:17,051 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=83, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/A/b67e4c73e8be43e9a0013560dc1049ce 2023-04-20 12:55:17,060 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/B/fa5432b2608947fd8551a336c40455ca 2023-04-20 12:55:17,068 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/C/3294920550ff484a8b148c448289d901 2023-04-20 12:55:17,074 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/A/b67e4c73e8be43e9a0013560dc1049ce as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/b67e4c73e8be43e9a0013560dc1049ce 2023-04-20 12:55:17,078 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/b67e4c73e8be43e9a0013560dc1049ce, entries=150, sequenceid=83, filesize=25.1 K 2023-04-20 12:55:17,079 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/B/fa5432b2608947fd8551a336c40455ca as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/fa5432b2608947fd8551a336c40455ca 2023-04-20 12:55:17,082 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/fa5432b2608947fd8551a336c40455ca, entries=150, sequenceid=83, filesize=11.6 K 2023-04-20 12:55:17,082 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/C/3294920550ff484a8b148c448289d901 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/3294920550ff484a8b148c448289d901 2023-04-20 12:55:17,086 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/3294920550ff484a8b148c448289d901, entries=150, sequenceid=83, filesize=11.6 K 2023-04-20 12:55:17,087 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=0 B/0 for aeecfead619437b2701aee4eba741c09 in 55ms, sequenceid=83, compaction requested=true 2023-04-20 12:55:17,087 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for aeecfead619437b2701aee4eba741c09: 2023-04-20 12:55:17,087 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:17,087 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:17,087 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:17,087 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,087 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:17,087 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:17,088 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,088 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:17,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:17,088 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,088 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:17,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:17,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:17,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:17,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:17,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:17,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4972b9ed[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:17,090 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:17,090 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,090 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,090 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:17,090 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:17,090 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:17,090 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,090 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:17,091 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:17,091 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:17,091 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:17,091 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:17,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:17,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:17,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:17,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:17,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:17,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:17,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:17,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:17,094 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:17,094 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:17,094 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:17,094 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:17,094 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:17,095 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,095 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,095 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:17,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:17,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:17,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:17,095 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:17,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:17,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:17,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:17,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:17,096 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:17,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:17,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,098 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,098 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:17,099 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:17,098 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:17,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:17,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:17,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:17,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:17,099 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:17,099 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:17,099 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:17,099 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:17,099 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,099 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,099 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,099 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,099 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:17,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:17,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:17,199 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:17,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:17,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:17,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:17,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:17,202 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:17,202 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:17,202 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:17,202 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:17,202 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,202 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:17,203 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,203 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:17,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:17,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:17,203 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,203 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:17,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:17,204 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:17,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:17,204 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:17,204 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:17,204 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:17,204 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:17,205 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:17,205 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:17,205 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:17,205 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:17,205 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:17,205 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:17,205 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for aeecfead619437b2701aee4eba741c09: 2023-04-20 12:55:17,205 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:17,205 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:17,205 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:17,205 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,205 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:17,205 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:17,223 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,223 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:17,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:17,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:17,224 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,224 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:17,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:17,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:17,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:17,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:17,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a3dd4eb[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:17,225 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:17,226 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,227 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,227 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:17,227 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,228 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:17,227 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:17,228 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:17,228 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:17,228 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:17,228 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:17,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:17,228 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:17,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:17,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:17,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:17,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:17,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:17,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:17,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:17,231 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:17,231 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:17,231 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:17,231 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:17,231 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:17,241 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/C/4bab9dee8c3a45cdb6958e797f410910 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/4bab9dee8c3a45cdb6958e797f410910 2023-04-20 12:55:17,244 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,244 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,244 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:17,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:17,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:17,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:17,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:17,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:17,245 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in aeecfead619437b2701aee4eba741c09/C of aeecfead619437b2701aee4eba741c09 into 4bab9dee8c3a45cdb6958e797f410910(size=11.7 K), total size for store is 34.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:55:17,245 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for aeecfead619437b2701aee4eba741c09: 2023-04-20 12:55:17,245 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09., storeName=aeecfead619437b2701aee4eba741c09/C, priority=13, startTime=1681995316393; duration=0sec 2023-04-20 12:55:17,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,245 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:55:17,245 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:17,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:17,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:17,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,247 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:17,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:17,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,252 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:17,253 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:17,253 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:17,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:17,253 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:17,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:17,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:17,253 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:17,253 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:17,253 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:17,253 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/A/42bdeca960674cac8294e9fcccc3a9a3 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/42bdeca960674cac8294e9fcccc3a9a3 2023-04-20 12:55:17,253 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,253 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,253 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,253 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,253 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:17,253 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:17,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:17,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:17,257 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in aeecfead619437b2701aee4eba741c09/A of aeecfead619437b2701aee4eba741c09 into 42bdeca960674cac8294e9fcccc3a9a3(size=25.2 K), total size for store is 75.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:55:17,257 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for aeecfead619437b2701aee4eba741c09: 2023-04-20 12:55:17,257 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09., storeName=aeecfead619437b2701aee4eba741c09/A, priority=13, startTime=1681995316393; duration=0sec 2023-04-20 12:55:17,257 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:55:17,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on aeecfead619437b2701aee4eba741c09 2023-04-20 12:55:17,334 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing aeecfead619437b2701aee4eba741c09 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-20 12:55:17,335 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=A 2023-04-20 12:55:17,335 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:17,335 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=B 2023-04-20 12:55:17,335 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:17,335 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=C 2023-04-20 12:55:17,335 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:17,353 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:17,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:17,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:17,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:17,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:17,357 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:17,357 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:17,357 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:17,357 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:17,359 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,359 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:17,360 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:17,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:17,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:17,360 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,361 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,361 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:17,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:17,362 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:17,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:17,362 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:17,370 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:17,370 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:17,370 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:17,370 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:17,370 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:17,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681995317361, max timestamp=1681995317380Num uniques 859; 2023-04-20 12:55:17,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681995317361, max timestamp=1681995317380Num uniques 859; 2023-04-20 12:55:17,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681995317361, max timestamp=1681995317380Num uniques 859; 2023-04-20 12:55:17,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-20 12:55:17,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-20 12:55:17,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-20 12:55:17,393 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:17,393 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:17,402 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:17,402 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:17,402 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:17,402 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:17,402 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:17,403 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:17,403 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,403 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:17,403 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:17,404 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:17,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:17,404 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:17,405 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,405 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:17,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:17,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:17,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:17,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:17,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7faa5de0[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:17,406 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:17,407 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,408 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304201cb25233b3b047be9fb1450d3f7f974f to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304201cb25233b3b047be9fb1450d3f7f974f 2023-04-20 12:55:17,408 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,408 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:17,411 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,411 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:17,411 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:17,411 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:17,413 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:17,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35892 deadline: 1681995377403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,415 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:17,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35842 deadline: 1681995377403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,415 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:17,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35900 deadline: 1681995377406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,419 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:17,419 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:17,419 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:17,419 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:17,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:17,420 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:17,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:17,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35886 deadline: 1681995377412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:17,421 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:17,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35848 deadline: 1681995377414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:17,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:17,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:17,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:17,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:17,428 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:17,428 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:17,428 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:17,428 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:17,428 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:17,429 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,429 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,429 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:17,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:17,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:17,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:17,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:17,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:17,429 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:17,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:17,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:17,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,431 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:17,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:17,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,444 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:17,444 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:17,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:17,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:17,444 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,444 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:17,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:17,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:17,444 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:17,445 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,445 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,445 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,445 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,445 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:17,445 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:17,521 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:17,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35892 deadline: 1681995377514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,527 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:17,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35842 deadline: 1681995377518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,527 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:17,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35900 deadline: 1681995377520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,527 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:17,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35886 deadline: 1681995377521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,528 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:17,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35848 deadline: 1681995377522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,545 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:17,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:17,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:17,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:17,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:17,562 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:17,562 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:17,563 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:17,563 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:17,563 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,563 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:17,564 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:17,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:17,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:17,564 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:17,564 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,564 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:17,565 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:17,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:17,565 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:17,565 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:17,566 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:17,566 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:17,566 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:17,566 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:17,578 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:17,578 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:17,578 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:17,578 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:17,578 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:17,578 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:17,578 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:17,578 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:17,578 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,579 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:17,579 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:17,580 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,580 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:17,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:17,580 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,580 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:17,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:17,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:17,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:17,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:17,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:17,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a4c8904[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:17,581 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:17,581 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,583 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,583 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:17,583 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:17,583 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:17,583 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,584 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:17,584 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,584 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:17,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:17,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:17,584 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:17,584 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:17,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:17,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:17,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:17,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:17,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:17,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:17,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:17,586 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:17,587 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:17,587 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:17,587 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:17,587 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:17,588 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,588 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:17,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:17,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:17,588 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:17,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:17,589 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:17,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:17,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:17,590 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:17,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:17,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:17,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,592 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:17,592 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:17,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:17,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:17,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:17,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:17,593 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,593 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:17,593 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:17,593 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,593 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,593 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,593 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,595 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:17,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:17,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:17,595 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:17,595 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:17,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:17,695 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:17,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:17,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:17,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:17,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:17,698 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:17,698 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:17,698 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:17,698 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:17,698 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,698 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:17,702 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,702 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:17,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:17,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:17,702 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,702 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:17,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:17,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:17,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:17,703 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:17,703 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:17,703 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:17,703 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:17,703 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:17,703 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:17,703 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:17,703 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:17,703 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:17,703 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:17,704 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:17,704 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:17,704 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:17,704 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:17,704 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,704 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:17,704 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:17,706 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:17,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:17,706 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:17,706 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,706 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:17,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:17,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:17,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:17,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:17,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16adfc71[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:17,707 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:17,707 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,708 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,708 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:17,708 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:17,708 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:17,708 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,708 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:17,708 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,708 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:17,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:17,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:17,708 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:17,709 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:17,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:17,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:17,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:17,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:17,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:17,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:17,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:17,711 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:17,711 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:17,711 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:17,711 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:17,711 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:17,713 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,713 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,713 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:17,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:17,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:17,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:17,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:17,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:17,714 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:17,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:17,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:17,714 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:17,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:17,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,716 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,716 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:17,716 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:17,716 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:17,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:17,716 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:17,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:17,716 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:17,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:17,716 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:17,716 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:17,716 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:17,716 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:17,716 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:17,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:17,716 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,716 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,727 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:17,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35892 deadline: 1681995377724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,731 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:17,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35842 deadline: 1681995377728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,731 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:17,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35886 deadline: 1681995377728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,735 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:17,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35900 deadline: 1681995377729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,735 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:17,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35848 deadline: 1681995377730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,816 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:17,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:17,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:17,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:17,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:17,819 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:17,819 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:17,819 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:17,819 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:17,820 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,820 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:17,820 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:17,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:17,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:17,820 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:17,821 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,821 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:17,821 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,821 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=96, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/A/d65522859f5e42f4b1f48d443f42831c 2023-04-20 12:55:17,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:17,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:17,822 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:17,822 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:17,822 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:17,822 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:17,822 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:17,822 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:17,826 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:17,826 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:17,827 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:17,827 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:17,827 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:17,827 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:17,827 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:17,828 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:17,828 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,828 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:17,828 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:17,836 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:17,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:17,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:17,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:17,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:17,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:17,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:17,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a3f8c3[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:17,838 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:17,838 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,838 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,840 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,840 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,840 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:17,840 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,840 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:17,840 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:17,840 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:17,840 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:17,841 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:17,841 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:17,841 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:17,842 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:17,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:17,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:17,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:17,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:17,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:17,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:17,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:17,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:17,845 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:17,845 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:17,845 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:17,845 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:17,845 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:17,848 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,848 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:17,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:17,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,848 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:17,848 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:17,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:17,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:17,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,850 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:17,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:17,852 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,852 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:17,852 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:17,852 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:17,852 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:17,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:17,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:17,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:17,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:17,852 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,852 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:17,852 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,852 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:17,852 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:17,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:17,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:17,852 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,852 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:17,853 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,853 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:17,953 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:17,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:17,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:17,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:17,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:17,955 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:17,955 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:17,955 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:17,955 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:17,956 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,956 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:17,956 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:17,956 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:17,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:17,957 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,957 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:17,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:17,957 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:17,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:17,957 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:17,958 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:17,958 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:17,958 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:17,958 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:17,958 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:17,958 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:17,958 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:17,958 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:17,958 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:17,958 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:17,958 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:17,958 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:17,958 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:17,958 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,958 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:17,958 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:17,959 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,959 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:17,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:17,959 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,959 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:17,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:17,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:17,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:17,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:17,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:17,960 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:17,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fcbe02a[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:17,960 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,961 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,961 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:17,961 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,961 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:17,961 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:17,961 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:17,962 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,962 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:17,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:17,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:17,962 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:17,962 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:17,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:17,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:17,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:17,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:17,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:17,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:17,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:17,964 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:17,964 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:17,964 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:17,964 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:17,964 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:17,966 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:17,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:17,966 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,966 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,966 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:17,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:17,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:17,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:17,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:17,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,968 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:17,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:17,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,969 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,969 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:17,969 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:17,969 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:17,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:17,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:17,969 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:17,969 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:17,969 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:17,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:17,969 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:17,969 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:17,969 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:17,970 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:17,970 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:17,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:17,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:17,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:17,974 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:18,034 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:18,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35892 deadline: 1681995378030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,036 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:18,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35886 deadline: 1681995378032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,036 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:18,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35842 deadline: 1681995378034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,043 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:18,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35848 deadline: 1681995378036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,043 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:18,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35900 deadline: 1681995378038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,070 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:18,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:18,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:18,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:18,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:18,072 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:18,073 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:18,073 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:18,073 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:18,073 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:18,073 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:18,073 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:18,074 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:18,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:18,074 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,074 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:18,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:18,074 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:18,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:18,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:18,074 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:18,074 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:18,075 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:18,075 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:18,075 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:18,075 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:18,075 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:18,075 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:18,075 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:18,075 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:18,075 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:18,075 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:18,075 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:18,075 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:18,075 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,075 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:18,075 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:18,076 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:18,076 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:18,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:18,076 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:18,076 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:18,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:18,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:18,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:18,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:18,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:18,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@161487e[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:18,077 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:18,077 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:18,078 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:18,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:18,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:18,079 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:18,079 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:18,079 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:18,079 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,079 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:18,079 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:18,079 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,079 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:18,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:18,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:18,079 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:18,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:18,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:18,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:18,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:18,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:18,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:18,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:18,081 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:18,081 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:18,081 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:18,081 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:18,081 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:18,082 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:18,082 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:18,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:18,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:18,082 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:18,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:18,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:18,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:18,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:18,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:18,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:18,082 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:18,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:18,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:18,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:18,083 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:18,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:18,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,085 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,085 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:18,085 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:18,085 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:18,085 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:18,085 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:18,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:18,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:18,085 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:18,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:18,085 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:18,085 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:18,085 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:18,085 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:18,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:18,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:18,085 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:18,085 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:18,185 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:18,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:18,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:18,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:18,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:18,188 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:18,188 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:18,188 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:18,188 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:18,188 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:18,188 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:18,189 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:18,189 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:18,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:18,189 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:18,189 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:18,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:18,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:18,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:18,190 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:18,190 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:18,190 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:18,190 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:18,190 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:18,190 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:18,190 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:18,190 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:18,190 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:18,190 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:18,190 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:18,190 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:18,190 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:18,190 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:18,190 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,190 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:18,190 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:18,191 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,191 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:18,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:18,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:18,191 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:18,191 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:18,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:18,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:18,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:18,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:18,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:18,192 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:18,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78d66257[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:18,192 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:18,193 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:18,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:18,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:18,193 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:18,193 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:18,193 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:18,193 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,193 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:18,194 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:18,194 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:18,194 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:18,194 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:18,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:18,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:18,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:18,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:18,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:18,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:18,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:18,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:18,196 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:18,196 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:18,196 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:18,196 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:18,196 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:18,197 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:18,197 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:18,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:18,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:18,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:18,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:18,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:18,197 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:18,197 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:18,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:18,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:18,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:18,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:18,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:18,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:18,198 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:18,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:18,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,199 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,199 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:18,199 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:18,199 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,199 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:18,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:18,199 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:18,199 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:18,199 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:18,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:18,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:18,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:18,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:18,199 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:18,199 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:18,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:18,200 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:18,200 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:18,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:18,231 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/B/342db4cb15b34a4a98415ecc6331bf66 2023-04-20 12:55:18,240 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/C/6d6f869f0ad14e2b8d58f3d0cf7da2a6 2023-04-20 12:55:18,243 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/A/d65522859f5e42f4b1f48d443f42831c as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/d65522859f5e42f4b1f48d443f42831c 2023-04-20 12:55:18,246 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/d65522859f5e42f4b1f48d443f42831c, entries=350, sequenceid=96, filesize=52.3 K 2023-04-20 12:55:18,247 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/B/342db4cb15b34a4a98415ecc6331bf66 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/342db4cb15b34a4a98415ecc6331bf66 2023-04-20 12:55:18,251 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/342db4cb15b34a4a98415ecc6331bf66, entries=150, sequenceid=96, filesize=11.6 K 2023-04-20 12:55:18,251 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/C/6d6f869f0ad14e2b8d58f3d0cf7da2a6 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/6d6f869f0ad14e2b8d58f3d0cf7da2a6 2023-04-20 12:55:18,254 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/6d6f869f0ad14e2b8d58f3d0cf7da2a6, entries=150, sequenceid=96, filesize=11.6 K 2023-04-20 12:55:18,255 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for aeecfead619437b2701aee4eba741c09 in 921ms, sequenceid=96, compaction requested=true 2023-04-20 12:55:18,255 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for aeecfead619437b2701aee4eba741c09: 2023-04-20 12:55:18,255 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:55:18,255 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 12:55:18,255 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 12:55:18,255 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:55:18,255 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:55:18,256 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 130851 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 12:55:18,256 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): aeecfead619437b2701aee4eba741c09/A is initiating minor compaction (all files) 2023-04-20 12:55:18,256 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47647 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 12:55:18,256 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of aeecfead619437b2701aee4eba741c09/A in TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:18,256 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): aeecfead619437b2701aee4eba741c09/B is initiating minor compaction (all files) 2023-04-20 12:55:18,256 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/42bdeca960674cac8294e9fcccc3a9a3, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/633e06da32f34a27bd5efbbdd68f5b39, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/b67e4c73e8be43e9a0013560dc1049ce, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/d65522859f5e42f4b1f48d443f42831c] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp, totalSize=127.8 K 2023-04-20 12:55:18,256 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of aeecfead619437b2701aee4eba741c09/B in TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:18,256 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/efc5aabd716e4153bd07fc63309dc66e, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/c6313b421da44e3ca49f62f6f4e7e8b6, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/fa5432b2608947fd8551a336c40455ca, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/342db4cb15b34a4a98415ecc6331bf66] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp, totalSize=46.5 K 2023-04-20 12:55:18,256 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 42bdeca960674cac8294e9fcccc3a9a3, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681995314478 2023-04-20 12:55:18,256 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting efc5aabd716e4153bd07fc63309dc66e, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681995314478 2023-04-20 12:55:18,257 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 633e06da32f34a27bd5efbbdd68f5b39, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1681995315127 2023-04-20 12:55:18,257 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting c6313b421da44e3ca49f62f6f4e7e8b6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1681995315127 2023-04-20 12:55:18,257 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting b67e4c73e8be43e9a0013560dc1049ce, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1681995316535 2023-04-20 12:55:18,257 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting fa5432b2608947fd8551a336c40455ca, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1681995316535 2023-04-20 12:55:18,257 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting d65522859f5e42f4b1f48d443f42831c, keycount=350, bloomtype=ROW, size=52.3 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1681995317244 2023-04-20 12:55:18,257 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 342db4cb15b34a4a98415ecc6331bf66, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1681995317320 2023-04-20 12:55:18,266 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): aeecfead619437b2701aee4eba741c09#B#compaction#323 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:55:18,270 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): aeecfead619437b2701aee4eba741c09#A#compaction#322 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:55:18,276 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/B/236a5c59b0ad45a69c391fff0c96276d as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/236a5c59b0ad45a69c391fff0c96276d 2023-04-20 12:55:18,279 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in aeecfead619437b2701aee4eba741c09/B of aeecfead619437b2701aee4eba741c09 into 236a5c59b0ad45a69c391fff0c96276d(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:55:18,280 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for aeecfead619437b2701aee4eba741c09: 2023-04-20 12:55:18,280 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09., storeName=aeecfead619437b2701aee4eba741c09/B, priority=12, startTime=1681995318255; duration=0sec 2023-04-20 12:55:18,280 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:55:18,280 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 12:55:18,281 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47647 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 12:55:18,281 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): aeecfead619437b2701aee4eba741c09/C is initiating minor compaction (all files) 2023-04-20 12:55:18,281 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of aeecfead619437b2701aee4eba741c09/C in TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:18,281 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/4bab9dee8c3a45cdb6958e797f410910, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/d7c0f9df03e4416c8f865b607caa0f45, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/3294920550ff484a8b148c448289d901, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/6d6f869f0ad14e2b8d58f3d0cf7da2a6] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp, totalSize=46.5 K 2023-04-20 12:55:18,281 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 4bab9dee8c3a45cdb6958e797f410910, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1681995314478 2023-04-20 12:55:18,281 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting d7c0f9df03e4416c8f865b607caa0f45, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1681995315127 2023-04-20 12:55:18,282 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 3294920550ff484a8b148c448289d901, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1681995316535 2023-04-20 12:55:18,282 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 6d6f869f0ad14e2b8d58f3d0cf7da2a6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1681995317320 2023-04-20 12:55:18,288 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): aeecfead619437b2701aee4eba741c09#C#compaction#324 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:55:18,300 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:18,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:18,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:18,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:18,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:18,303 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:18,303 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:18,303 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:18,303 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:18,303 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:18,303 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:18,304 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,304 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:18,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:18,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:18,304 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,304 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:18,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:18,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:18,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:18,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:18,304 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:18,305 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:18,305 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:18,305 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:18,305 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:18,305 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:18,306 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:18,306 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:18,306 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:18,306 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:18,306 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing aeecfead619437b2701aee4eba741c09 3/3 column families, dataSize=140.89 KB heapSize=386.16 KB 2023-04-20 12:55:18,308 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=A 2023-04-20 12:55:18,308 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:55:18,308 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=B 2023-04-20 12:55:18,308 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:55:18,308 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=C 2023-04-20 12:55:18,308 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:55:18,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on aeecfead619437b2701aee4eba741c09 2023-04-20 12:55:18,544 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:18,554 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:18,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35892 deadline: 1681995378552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,560 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:18,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35848 deadline: 1681995378553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,565 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:18,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35842 deadline: 1681995378554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,565 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:18,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35900 deadline: 1681995378554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,566 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:18,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35886 deadline: 1681995378554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,658 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:18,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35892 deadline: 1681995378655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,664 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:18,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35848 deadline: 1681995378661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,670 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:18,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35842 deadline: 1681995378666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,671 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:18,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35900 deadline: 1681995378666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,671 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:18,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35886 deadline: 1681995378666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,699 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/C/a0721b80402b4bd59df816c3a8950731 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/a0721b80402b4bd59df816c3a8950731 2023-04-20 12:55:18,703 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in aeecfead619437b2701aee4eba741c09/C of aeecfead619437b2701aee4eba741c09 into a0721b80402b4bd59df816c3a8950731(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:55:18,703 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for aeecfead619437b2701aee4eba741c09: 2023-04-20 12:55:18,703 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09., storeName=aeecfead619437b2701aee4eba741c09/C, priority=12, startTime=1681995318255; duration=0sec 2023-04-20 12:55:18,703 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:55:18,727 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304209ad01c18305f483a8fc751543efafea2 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304209ad01c18305f483a8fc751543efafea2 2023-04-20 12:55:18,863 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:18,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35892 deadline: 1681995378859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,871 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:18,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35848 deadline: 1681995378866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,876 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:18,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35842 deadline: 1681995378871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,878 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:18,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35886 deadline: 1681995378873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:18,878 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:18,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35900 deadline: 1681995378873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,131 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=131, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/A/c1ce1022ac414922971a4f264ddf8317 2023-04-20 12:55:19,139 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=131 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/B/adcf302345f84faea891e9565e0f377a 2023-04-20 12:55:19,149 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=131 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/C/5acdc459d92e40b291c6a4b0ab067365 2023-04-20 12:55:19,153 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/A/c1ce1022ac414922971a4f264ddf8317 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/c1ce1022ac414922971a4f264ddf8317 2023-04-20 12:55:19,157 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/c1ce1022ac414922971a4f264ddf8317, entries=150, sequenceid=131, filesize=25.2 K 2023-04-20 12:55:19,158 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/B/adcf302345f84faea891e9565e0f377a as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/adcf302345f84faea891e9565e0f377a 2023-04-20 12:55:19,161 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/adcf302345f84faea891e9565e0f377a, entries=150, sequenceid=131, filesize=11.7 K 2023-04-20 12:55:19,170 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/C/5acdc459d92e40b291c6a4b0ab067365 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/5acdc459d92e40b291c6a4b0ab067365 2023-04-20 12:55:19,172 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:19,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35892 deadline: 1681995379164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,174 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/5acdc459d92e40b291c6a4b0ab067365, entries=150, sequenceid=131, filesize=11.7 K 2023-04-20 12:55:19,174 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-20 12:55:19,174 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~386.11 KB/395376, currentSize=53.67 KB/54960 for aeecfead619437b2701aee4eba741c09 in 868ms, sequenceid=131, compaction requested=false 2023-04-20 12:55:19,175 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for aeecfead619437b2701aee4eba741c09: 2023-04-20 12:55:19,175 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:19,175 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:19,175 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:19,175 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,175 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:19,175 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:19,176 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,176 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:19,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:19,176 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:19,176 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:19,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:19,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on aeecfead619437b2701aee4eba741c09 2023-04-20 12:55:19,177 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing aeecfead619437b2701aee4eba741c09 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-20 12:55:19,177 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=A 2023-04-20 12:55:19,177 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:19,177 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=B 2023-04-20 12:55:19,177 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:19,177 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=C 2023-04-20 12:55:19,177 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:19,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:19,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:19,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:19,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65aee86e[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:19,178 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:19,178 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,179 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,179 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:19,179 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:19,179 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:19,180 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,180 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:19,180 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:19,180 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:19,180 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,180 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:19,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:19,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:19,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:19,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:19,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:19,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:19,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:19,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:19,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:19,182 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:19,182 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:19,182 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:19,183 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:19,183 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:19,184 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,184 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:19,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:19,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,184 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:19,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:19,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:19,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:19,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,185 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:19,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:19,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:19,186 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,188 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:19,188 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:19,188 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:19,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:19,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:19,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:19,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:19,188 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:19,188 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:19,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:19,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:19,190 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,190 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:19,190 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:19,190 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,190 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,190 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,190 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:19,193 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:19,217 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:19,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35886 deadline: 1681995379214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,221 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:19,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35842 deadline: 1681995379215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,221 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:19,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35900 deadline: 1681995379215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,221 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:19,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35848 deadline: 1681995379216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,288 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:19,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:19,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:19,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:19,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:19,290 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:19,291 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:19,291 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:19,291 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:19,291 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,291 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:19,292 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,292 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:19,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:19,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:19,292 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,292 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:19,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:19,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:19,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:19,293 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:19,293 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:19,293 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:19,293 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:19,293 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:19,293 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:19,293 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:19,293 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:19,293 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:19,293 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:19,293 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:19,293 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:19,293 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:19,293 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:19,293 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,293 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:19,293 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:19,294 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,294 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:19,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:19,294 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,294 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:19,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:19,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:19,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:19,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:19,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:19,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15149e41[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:19,295 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:19,295 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,296 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,296 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,296 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:19,296 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:19,296 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:19,296 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:19,298 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:19,298 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,298 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:19,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:19,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:19,298 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:19,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:19,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:19,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:19,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:19,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:19,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:19,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:19,299 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:19,300 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:19,300 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:19,300 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:19,300 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:19,300 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,300 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,300 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:19,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:19,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:19,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,301 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:19,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:19,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:19,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:19,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:19,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:19,302 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:19,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:19,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,303 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,303 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:19,303 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:19,303 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:19,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:19,303 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:19,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:19,303 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:19,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:19,303 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:19,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:19,304 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,304 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,304 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:19,304 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,304 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:19,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:19,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:19,321 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:19,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35886 deadline: 1681995379318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,325 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:19,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35842 deadline: 1681995379322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,325 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:19,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35900 deadline: 1681995379322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,325 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:19,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35848 deadline: 1681995379322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,404 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:19,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:19,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:19,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:19,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:19,407 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:19,407 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:19,407 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:19,407 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:19,407 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,407 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:19,408 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:19,408 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:19,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:19,408 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,408 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:19,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:19,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:19,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:19,409 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:19,409 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:19,409 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:19,409 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:19,409 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:19,409 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:19,409 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:19,409 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:19,409 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:19,409 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:19,410 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:19,410 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:19,410 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:19,410 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:19,410 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,410 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:19,410 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:19,411 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,411 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:19,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:19,411 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,411 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:19,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:19,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:19,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:19,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:19,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:19,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ebd0625[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:19,412 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:19,412 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,413 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,413 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:19,413 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:19,413 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:19,413 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,413 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:19,414 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:19,414 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:19,414 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,414 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:19,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:19,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:19,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:19,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:19,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:19,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:19,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:19,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:19,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:19,417 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:19,417 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:19,417 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:19,417 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:19,417 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:19,418 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,418 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:19,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,418 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:19,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:19,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:19,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:19,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:19,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:19,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:19,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:19,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,420 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:19,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:19,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,421 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,422 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:19,422 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:19,422 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:19,422 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:19,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:19,422 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:19,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:19,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:19,422 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:19,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:19,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:19,423 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,423 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,423 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:19,423 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,423 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:19,423 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,487 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/A/25998a9af000484a878b4f7e424ff560 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/25998a9af000484a878b4f7e424ff560 2023-04-20 12:55:19,490 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in aeecfead619437b2701aee4eba741c09/A of aeecfead619437b2701aee4eba741c09 into 25998a9af000484a878b4f7e424ff560(size=25.4 K), total size for store is 50.6 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-20 12:55:19,491 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for aeecfead619437b2701aee4eba741c09: 2023-04-20 12:55:19,491 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09., storeName=aeecfead619437b2701aee4eba741c09/A, priority=12, startTime=1681995318255; duration=1sec 2023-04-20 12:55:19,491 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:55:19,523 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:19,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:19,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:19,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:19,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:19,526 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:19,526 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:19,526 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:19,526 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:19,527 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,527 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:19,527 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:19,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35886 deadline: 1681995379523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,527 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:19,527 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:19,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:19,528 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,528 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:19,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:19,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:19,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:19,528 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:19,529 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:19,529 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:19,529 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:19,529 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:19,529 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:19,529 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:19,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35900 deadline: 1681995379526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,529 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:19,529 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:19,529 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:19,529 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:19,529 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:19,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35842 deadline: 1681995379526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,529 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:19,529 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:19,529 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:19,530 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:19,530 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,530 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:19,530 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:19,530 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,530 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,530 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:19,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:19,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:19,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35848 deadline: 1681995379527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,531 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,531 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:19,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:19,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:19,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:19,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:19,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:19,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55b20b7f[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:19,532 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:19,532 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,532 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,533 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:19,533 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:19,533 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:19,533 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,533 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:19,533 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,533 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:19,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:19,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:19,533 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:19,533 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:19,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:19,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:19,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:19,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:19,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:19,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:19,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:19,536 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:19,536 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:19,536 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:19,536 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:19,536 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:19,537 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,537 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:19,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:19,537 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:19,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:19,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:19,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:19,537 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:19,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:19,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:19,538 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:19,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:19,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,539 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:19,539 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:19,539 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:19,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:19,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:19,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:19,539 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:19,540 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:19,540 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,540 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,540 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,540 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:19,540 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:19,539 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:19,540 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:19,540 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:19,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:19,590 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042044cd7ff2c2e2415495a0b407a7c4d1c4 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042044cd7ff2c2e2415495a0b407a7c4d1c4 2023-04-20 12:55:19,640 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:19,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:19,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:19,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:19,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:19,654 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:19,655 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:19,655 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:19,655 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:19,655 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,655 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:19,656 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,656 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:19,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:19,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:19,656 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,656 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:19,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:19,656 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:19,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:19,657 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:19,657 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:19,657 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:19,657 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:19,657 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:19,657 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:19,657 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:19,657 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:19,657 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:19,657 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:19,657 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:19,657 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:19,657 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:19,657 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:19,657 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,657 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:19,657 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:19,658 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,658 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:19,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:19,659 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,659 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:19,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:19,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:19,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:19,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:19,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:19,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@474d743f[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:19,660 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:19,660 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,660 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,660 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,660 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:19,660 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:19,660 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:19,660 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:19,661 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:19,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:19,661 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:19,661 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:19,661 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:19,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:19,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:19,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:19,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:19,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:19,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:19,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:19,664 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:19,664 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:19,664 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:19,664 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:19,664 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:19,665 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,665 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,665 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:19,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:19,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:19,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,666 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:19,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:19,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:19,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:19,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:19,666 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:19,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:19,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:19,668 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,668 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:19,668 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:19,668 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,669 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,669 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:19,669 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:19,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:19,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:19,669 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:19,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:19,669 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,669 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:19,669 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:19,669 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:19,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:19,669 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:19,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:19,684 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:19,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35892 deadline: 1681995379679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,769 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:19,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:19,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:19,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:19,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:19,772 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:19,772 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:19,772 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:19,772 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:19,772 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,772 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:19,773 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:19,773 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:19,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:19,773 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,773 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:19,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:19,773 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:19,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:19,774 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:19,774 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:19,774 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:19,774 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:19,774 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:19,774 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:19,774 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:19,774 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:19,774 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:19,774 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:19,774 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:19,774 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:19,774 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:19,774 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:19,774 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,774 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:19,774 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:19,775 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,775 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:19,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:19,775 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,775 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:19,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:19,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:19,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:19,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:19,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:19,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@761a0c01[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:19,777 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:19,777 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,777 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,777 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:19,778 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,778 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:19,778 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:19,778 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:19,779 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:19,780 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,780 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:19,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:19,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:19,780 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:19,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:19,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:19,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:19,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:19,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:19,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:19,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:19,781 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:19,781 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:19,781 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:19,782 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:19,782 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:19,782 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,782 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:19,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:19,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,782 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:19,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:19,783 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:19,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:19,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:19,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:19,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:19,785 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:19,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:19,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,787 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,787 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:19,787 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:19,787 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:19,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:19,787 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:19,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:19,787 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:19,787 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:19,787 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,787 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:19,787 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:19,787 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:19,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:19,788 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:19,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:19,833 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:19,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35900 deadline: 1681995379830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,834 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:19,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35842 deadline: 1681995379831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,834 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:19,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35848 deadline: 1681995379832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,834 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:19,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35886 deadline: 1681995379830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,887 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:19,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:19,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:19,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:19,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:19,891 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:19,891 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:19,891 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:19,891 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:19,892 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,892 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:19,892 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,892 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:19,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:19,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:19,893 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,893 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:19,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:19,893 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:19,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:19,893 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:19,893 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:19,894 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:19,894 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:19,894 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:19,894 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:19,894 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:19,894 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:19,894 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:19,894 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:19,894 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:19,894 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:19,894 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:19,894 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:19,894 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,894 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:19,894 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:19,895 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,895 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:19,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:19,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:19,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:19,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:19,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:19,896 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:19,897 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:19,897 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:19,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e0b0c9[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:19,897 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,897 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,897 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:19,897 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:19,897 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:19,897 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,897 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:19,898 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,898 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:19,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:19,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:19,898 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:19,898 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:19,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:19,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:19,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:19,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:19,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:19,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:19,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:19,900 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:19,900 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:19,900 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:19,900 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:19,900 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:19,901 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,901 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:19,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:19,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:19,901 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:19,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:19,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:19,902 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:19,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:19,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:19,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,903 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:19,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:19,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,905 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,905 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:19,905 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:19,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:19,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:19,905 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:19,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:19,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:19,905 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:19,905 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:19,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:19,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:19,905 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:19,905 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:19,905 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,905 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:19,905 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:19,905 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:19,995 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=144, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/A/0d9970eff125410d9f7259568da65a3f 2023-04-20 12:55:20,005 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:20,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:20,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:20,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:20,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:20,010 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:20,010 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:20,010 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:20,010 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:20,011 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:20,011 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:20,011 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:20,011 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:20,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:20,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:20,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:20,012 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:20,012 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:20,012 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:20,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:20,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:20,013 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:20,013 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:20,013 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:20,013 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:20,013 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:20,013 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:20,013 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:20,013 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:20,013 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:20,013 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:20,013 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:20,013 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:20,013 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:20,013 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:20,013 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:20,013 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:20,013 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:20,014 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=144 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/B/618176180014400ab694fe6d0a86adf8 2023-04-20 12:55:20,014 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:20,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:20,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:20,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:20,014 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:20,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:20,015 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:20,015 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:20,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:20,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:20,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:20,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:20,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:20,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45ccba59[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:20,016 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:20,016 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:20,017 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:20,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:20,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:20,017 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:20,017 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:20,017 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:20,017 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:20,017 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:20,018 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:20,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:20,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:20,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:20,019 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:20,019 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:20,019 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:20,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:20,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:20,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:20,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:20,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:20,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:20,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:20,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:20,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:20,022 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:20,022 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:20,022 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:20,022 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:20,022 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:20,024 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:20,024 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:20,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:20,024 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:20,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:20,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:20,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:20,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:20,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:20,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:20,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:20,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:20,025 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:20,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:20,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:20,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:20,027 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:20,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:20,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:20,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:20,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:20,037 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:20,037 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:20,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:20,037 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:20,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:20,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:20,037 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:20,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:20,037 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:20,037 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:20,037 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:20,037 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:20,037 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:20,037 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:20,038 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:20,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:20,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:20,038 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:20,038 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:20,064 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=144 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/C/e8df2ce2b8d443249dac7fee27fb8157 2023-04-20 12:55:20,068 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/A/0d9970eff125410d9f7259568da65a3f as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/0d9970eff125410d9f7259568da65a3f 2023-04-20 12:55:20,074 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/0d9970eff125410d9f7259568da65a3f, entries=200, sequenceid=144, filesize=32.1 K 2023-04-20 12:55:20,075 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/B/618176180014400ab694fe6d0a86adf8 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/618176180014400ab694fe6d0a86adf8 2023-04-20 12:55:20,081 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/618176180014400ab694fe6d0a86adf8, entries=150, sequenceid=144, filesize=11.8 K 2023-04-20 12:55:20,083 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/C/e8df2ce2b8d443249dac7fee27fb8157 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/e8df2ce2b8d443249dac7fee27fb8157 2023-04-20 12:55:20,088 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/e8df2ce2b8d443249dac7fee27fb8157, entries=150, sequenceid=144, filesize=11.8 K 2023-04-20 12:55:20,089 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for aeecfead619437b2701aee4eba741c09 in 912ms, sequenceid=144, compaction requested=true 2023-04-20 12:55:20,089 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for aeecfead619437b2701aee4eba741c09: 2023-04-20 12:55:20,089 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:55:20,089 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:55:20,089 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:55:20,089 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:55:20,089 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:55:20,090 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84668 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:55:20,090 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36098 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:55:20,090 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): aeecfead619437b2701aee4eba741c09/A is initiating minor compaction (all files) 2023-04-20 12:55:20,090 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): aeecfead619437b2701aee4eba741c09/B is initiating minor compaction (all files) 2023-04-20 12:55:20,090 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of aeecfead619437b2701aee4eba741c09/A in TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:20,090 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of aeecfead619437b2701aee4eba741c09/B in TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:20,090 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/25998a9af000484a878b4f7e424ff560, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/c1ce1022ac414922971a4f264ddf8317, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/0d9970eff125410d9f7259568da65a3f] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp, totalSize=82.7 K 2023-04-20 12:55:20,090 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/236a5c59b0ad45a69c391fff0c96276d, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/adcf302345f84faea891e9565e0f377a, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/618176180014400ab694fe6d0a86adf8] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp, totalSize=35.3 K 2023-04-20 12:55:20,091 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 25998a9af000484a878b4f7e424ff560, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1681995317320 2023-04-20 12:55:20,091 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 236a5c59b0ad45a69c391fff0c96276d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1681995317320 2023-04-20 12:55:20,091 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting c1ce1022ac414922971a4f264ddf8317, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1681995317402 2023-04-20 12:55:20,091 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting adcf302345f84faea891e9565e0f377a, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1681995317402 2023-04-20 12:55:20,092 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 0d9970eff125410d9f7259568da65a3f, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1681995318552 2023-04-20 12:55:20,092 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 618176180014400ab694fe6d0a86adf8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1681995318553 2023-04-20 12:55:20,101 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): aeecfead619437b2701aee4eba741c09#B#compaction#331 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:55:20,127 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): aeecfead619437b2701aee4eba741c09#A#compaction#332 average throughput is 0.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:55:20,134 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/B/70eb4f2936b647dea43220522ef33d65 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/70eb4f2936b647dea43220522ef33d65 2023-04-20 12:55:20,138 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:20,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:20,141 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in aeecfead619437b2701aee4eba741c09/B of aeecfead619437b2701aee4eba741c09 into 70eb4f2936b647dea43220522ef33d65(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:55:20,141 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for aeecfead619437b2701aee4eba741c09: 2023-04-20 12:55:20,141 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09., storeName=aeecfead619437b2701aee4eba741c09/B, priority=13, startTime=1681995320089; duration=0sec 2023-04-20 12:55:20,141 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:55:20,141 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:55:20,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:20,142 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36098 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:55:20,142 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): aeecfead619437b2701aee4eba741c09/C is initiating minor compaction (all files) 2023-04-20 12:55:20,142 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of aeecfead619437b2701aee4eba741c09/C in TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:20,142 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/a0721b80402b4bd59df816c3a8950731, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/5acdc459d92e40b291c6a4b0ab067365, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/e8df2ce2b8d443249dac7fee27fb8157] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp, totalSize=35.3 K 2023-04-20 12:55:20,143 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting a0721b80402b4bd59df816c3a8950731, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1681995317320 2023-04-20 12:55:20,144 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 5acdc459d92e40b291c6a4b0ab067365, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1681995317402 2023-04-20 12:55:20,145 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting e8df2ce2b8d443249dac7fee27fb8157, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1681995318553 2023-04-20 12:55:20,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:20,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:20,163 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:20,163 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:20,163 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:20,163 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:20,163 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:20,163 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:20,164 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:20,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:20,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:20,164 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:20,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:20,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:20,165 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:20,165 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:20,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:20,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:20,165 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:20,170 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:20,170 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:20,170 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:20,170 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:20,170 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:20,172 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): aeecfead619437b2701aee4eba741c09#C#compaction#333 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:55:20,174 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:20,174 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:20,174 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:20,174 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:20,175 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing aeecfead619437b2701aee4eba741c09 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-20 12:55:20,175 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=A 2023-04-20 12:55:20,175 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:20,175 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=B 2023-04-20 12:55:20,175 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:20,175 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=C 2023-04-20 12:55:20,175 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:20,237 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/C/c1f39d61f63b44068846f4978f1764c4 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/c1f39d61f63b44068846f4978f1764c4 2023-04-20 12:55:20,244 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in aeecfead619437b2701aee4eba741c09/C of aeecfead619437b2701aee4eba741c09 into c1f39d61f63b44068846f4978f1764c4(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:55:20,245 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for aeecfead619437b2701aee4eba741c09: 2023-04-20 12:55:20,245 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09., storeName=aeecfead619437b2701aee4eba741c09/C, priority=13, startTime=1681995320089; duration=0sec 2023-04-20 12:55:20,245 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:55:20,342 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:20,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on aeecfead619437b2701aee4eba741c09 2023-04-20 12:55:20,355 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:20,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35886 deadline: 1681995380348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:20,361 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:20,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35900 deadline: 1681995380353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:20,362 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:20,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35842 deadline: 1681995380354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:20,362 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:20,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35848 deadline: 1681995380355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:20,458 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:20,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35886 deadline: 1681995380456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:20,467 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:20,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35900 deadline: 1681995380464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:20,467 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:20,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35842 deadline: 1681995380464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:20,468 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:20,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35848 deadline: 1681995380465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:20,642 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420bbef27b98ed443e48a0c525d79ceabc2 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420bbef27b98ed443e48a0c525d79ceabc2 2023-04-20 12:55:20,655 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/A/d1bdd99960734e19a03f7f0b2a311120 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/d1bdd99960734e19a03f7f0b2a311120 2023-04-20 12:55:20,658 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in aeecfead619437b2701aee4eba741c09/A of aeecfead619437b2701aee4eba741c09 into d1bdd99960734e19a03f7f0b2a311120(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:55:20,658 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for aeecfead619437b2701aee4eba741c09: 2023-04-20 12:55:20,658 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09., storeName=aeecfead619437b2701aee4eba741c09/A, priority=13, startTime=1681995320089; duration=0sec 2023-04-20 12:55:20,658 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:55:20,664 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:20,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35886 deadline: 1681995380661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:20,671 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:20,672 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:20,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35900 deadline: 1681995380668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:20,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35848 deadline: 1681995380668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:20,672 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:20,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35842 deadline: 1681995380669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:20,692 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:20,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35892 deadline: 1681995380690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:20,970 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:20,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35886 deadline: 1681995380966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:20,976 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:20,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35842 deadline: 1681995380973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:20,977 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:20,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35900 deadline: 1681995380974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:20,977 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:20,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35848 deadline: 1681995380975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,046 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=166, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/A/6517274f9a5d4c37a48ac0c3c50042d4 2023-04-20 12:55:21,456 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=166 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/B/ec6d7cd665c749acb0b0b4b324a52f5a 2023-04-20 12:55:21,463 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=166 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/C/5d3f4c05b20d4528a9a0d17f013af497 2023-04-20 12:55:21,466 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/A/6517274f9a5d4c37a48ac0c3c50042d4 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/6517274f9a5d4c37a48ac0c3c50042d4 2023-04-20 12:55:21,468 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/6517274f9a5d4c37a48ac0c3c50042d4, entries=150, sequenceid=166, filesize=25.3 K 2023-04-20 12:55:21,469 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/B/ec6d7cd665c749acb0b0b4b324a52f5a as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/ec6d7cd665c749acb0b0b4b324a52f5a 2023-04-20 12:55:21,473 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:21,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35886 deadline: 1681995381472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,480 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:21,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35900 deadline: 1681995381478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,482 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:21,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35848 deadline: 1681995381479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,486 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:21,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35842 deadline: 1681995381481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,487 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/ec6d7cd665c749acb0b0b4b324a52f5a, entries=150, sequenceid=166, filesize=11.8 K 2023-04-20 12:55:21,488 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/C/5d3f4c05b20d4528a9a0d17f013af497 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/5d3f4c05b20d4528a9a0d17f013af497 2023-04-20 12:55:21,491 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/5d3f4c05b20d4528a9a0d17f013af497, entries=150, sequenceid=166, filesize=11.8 K 2023-04-20 12:55:21,492 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for aeecfead619437b2701aee4eba741c09 in 1317ms, sequenceid=166, compaction requested=false 2023-04-20 12:55:21,492 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for aeecfead619437b2701aee4eba741c09: 2023-04-20 12:55:21,492 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:21,492 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:21,492 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:21,492 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,492 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:21,492 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:21,514 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,514 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:21,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:21,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:21,515 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:21,515 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:21,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:21,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:21,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:21,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:21,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:21,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@690cd253[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:21,516 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:21,516 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:21,565 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:21,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:21,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:21,565 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,565 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:21,565 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:21,565 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:21,565 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:21,566 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:21,566 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,566 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:21,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:21,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:21,566 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:21,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:21,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:21,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:21,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:21,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:21,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:21,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:21,568 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:21,568 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:21,568 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:21,568 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:21,568 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:21,569 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:21,569 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:21,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:21,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:21,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:21,569 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:21,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:21,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:21,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:21,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:21,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:21,570 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:21,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:21,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:21,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:21,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,571 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:21,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:21,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,572 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,572 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:21,572 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:21,572 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:21,572 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:21,572 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,572 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:21,572 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:21,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:21,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:21,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:21,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:21,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:21,572 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:21,573 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:21,572 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:21,573 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:21,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:21,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:21,673 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:21,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:21,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:21,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:21,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:21,676 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:21,676 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:21,676 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:21,676 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:21,676 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:21,676 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:21,677 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,677 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:21,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:21,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:21,677 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,677 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:21,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:21,678 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:21,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:21,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:21,678 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:21,678 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:21,678 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:21,678 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:21,678 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:21,678 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:21,679 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:21,679 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:21,679 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:21,679 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:21,679 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing aeecfead619437b2701aee4eba741c09 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-20 12:55:21,679 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=A 2023-04-20 12:55:21,679 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:21,679 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=B 2023-04-20 12:55:21,679 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:21,679 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=C 2023-04-20 12:55:21,679 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:21,700 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304208412f6e5c7f043f9b1c2f23fffae5dda to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304208412f6e5c7f043f9b1c2f23fffae5dda 2023-04-20 12:55:21,715 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=180, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/A/0261f2f8bf934ef49e53cd0b795110ab 2023-04-20 12:55:21,735 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=180 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/B/5b7415cb7795431282c6834f9b0b741e 2023-04-20 12:55:21,746 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=180 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/C/8169a02f389446f7a783465051126740 2023-04-20 12:55:21,750 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/A/0261f2f8bf934ef49e53cd0b795110ab as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/0261f2f8bf934ef49e53cd0b795110ab 2023-04-20 12:55:21,753 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/0261f2f8bf934ef49e53cd0b795110ab, entries=150, sequenceid=180, filesize=25.3 K 2023-04-20 12:55:21,754 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/B/5b7415cb7795431282c6834f9b0b741e as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/5b7415cb7795431282c6834f9b0b741e 2023-04-20 12:55:21,758 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/5b7415cb7795431282c6834f9b0b741e, entries=150, sequenceid=180, filesize=11.8 K 2023-04-20 12:55:21,759 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/C/8169a02f389446f7a783465051126740 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/8169a02f389446f7a783465051126740 2023-04-20 12:55:21,764 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/8169a02f389446f7a783465051126740, entries=150, sequenceid=180, filesize=11.8 K 2023-04-20 12:55:21,765 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for aeecfead619437b2701aee4eba741c09 in 86ms, sequenceid=180, compaction requested=true 2023-04-20 12:55:21,765 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for aeecfead619437b2701aee4eba741c09: 2023-04-20 12:55:21,765 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:21,765 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:21,765 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:21,765 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,765 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:21,765 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:21,766 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,766 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:21,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:21,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:21,767 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:21,767 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:21,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:21,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:21,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:21,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:21,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:21,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f50a177[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:21,768 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:21,768 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:21,772 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:21,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:21,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:21,772 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:21,772 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:21,773 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:21,772 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,773 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:21,773 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,773 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:21,773 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:21,773 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:21,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:21,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:21,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:21,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:21,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:21,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:21,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:21,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:21,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:21,776 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:21,776 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:21,776 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:21,776 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:21,776 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:21,777 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:21,777 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:21,777 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:21,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:21,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:21,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:21,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:21,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:21,778 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:21,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:21,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:21,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:21,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:21,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:21,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:21,779 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:21,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:21,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,780 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:21,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:21,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:21,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:21,780 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:21,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:21,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:21,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:21,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:21,780 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:21,781 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:21,781 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:21,780 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,781 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:21,781 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:21,781 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,781 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:21,781 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:21,781 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:21,881 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:21,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:21,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:21,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:21,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:21,886 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:21,886 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:21,886 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:21,886 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:21,886 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:21,886 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:21,887 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,887 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:21,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:21,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:21,887 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,887 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:21,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:21,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:21,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:21,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:21,888 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:21,888 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:21,888 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:21,888 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:21,889 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:21,889 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:21,910 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:21,910 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:21,910 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:21,910 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:21,910 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for aeecfead619437b2701aee4eba741c09: 2023-04-20 12:55:21,910 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:21,910 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:21,910 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:21,910 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,910 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:21,911 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:21,912 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,912 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:21,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:21,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:21,912 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:21,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:21,912 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:21,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:21,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:21,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:21,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:21,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ab336ac[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:21,914 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:21,914 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:21,914 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:21,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:21,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:21,914 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:21,914 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,914 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:21,914 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:21,915 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:21,915 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:21,915 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,915 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:21,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:21,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:21,916 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:21,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:21,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:21,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:21,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:21,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:21,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:21,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:21,918 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:21,918 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:21,918 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:21,918 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:21,918 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:21,919 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:21,919 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:21,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:21,919 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:21,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:21,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:21,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:21,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:21,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:21,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:21,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:21,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:21,922 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:21,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:21,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:21,923 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:21,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:21,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,932 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,932 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:21,932 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:21,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:21,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:21,932 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:21,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:21,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:21,932 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:21,932 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:21,932 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:21,932 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:21,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:21,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:21,932 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:21,932 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:21,933 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:21,933 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:22,033 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:22,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:22,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:22,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:22,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:22,036 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:22,036 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:22,036 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:22,036 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:22,037 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:22,037 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:22,038 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:22,038 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:22,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:22,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:22,039 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,039 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:22,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:22,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:22,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:22,040 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:22,040 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:22,040 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:22,040 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:22,040 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:22,040 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:22,040 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:22,040 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:22,040 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:22,040 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:22,041 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for aeecfead619437b2701aee4eba741c09: 2023-04-20 12:55:22,041 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:22,041 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:22,041 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:22,041 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,041 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:22,041 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:22,042 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:22,042 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:22,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:22,042 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:22,042 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:22,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:22,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:22,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:22,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:22,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:22,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a8d43d1[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:22,043 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:22,043 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:22,044 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:22,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:22,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:22,044 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:22,044 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:22,044 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:22,045 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,045 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:22,045 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,045 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:22,045 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:22,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:22,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:22,045 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:22,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:22,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:22,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:22,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:22,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:22,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:22,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:22,047 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:22,047 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:22,047 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:22,047 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:22,048 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:22,049 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:22,049 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:22,049 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:22,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:22,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:22,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:22,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:22,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:22,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:22,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:22,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:22,049 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:22,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:22,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:22,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:22,050 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:22,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:22,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,052 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,052 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:22,052 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:22,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:22,052 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:22,052 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:22,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:22,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:22,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:22,052 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:22,052 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,052 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:22,052 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:22,052 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:22,053 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:22,053 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:22,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:22,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:22,053 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:22,153 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:22,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:22,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:22,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:22,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:22,156 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:22,156 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:22,156 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:22,156 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:22,156 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:22,156 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:22,157 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:22,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:22,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:22,157 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:22,157 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,157 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:22,158 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:22,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:22,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:22,158 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:22,158 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:22,158 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:22,158 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:22,158 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:22,158 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:22,158 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:22,158 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:22,158 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:22,159 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:22,159 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for aeecfead619437b2701aee4eba741c09: 2023-04-20 12:55:22,159 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:22,159 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:22,159 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:22,159 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,160 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:22,160 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:22,160 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:22,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:22,161 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:22,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:22,161 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:22,161 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:22,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:22,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:22,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:22,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:22,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@651c8a99[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:22,163 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:22,163 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:22,163 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:22,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:22,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:22,163 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:22,163 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:22,163 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:22,164 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,164 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:22,164 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:22,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:22,164 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:22,164 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:22,165 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:22,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:22,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:22,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:22,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:22,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:22,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:22,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:22,167 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:22,167 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:22,167 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:22,167 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:22,167 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:22,168 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:22,168 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:22,168 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:22,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:22,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:22,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:22,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:22,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:22,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:22,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:22,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:22,169 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:22,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:22,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:22,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:22,170 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:22,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:22,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,172 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,172 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:22,172 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:22,172 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:22,172 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:22,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:22,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:22,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:22,172 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:22,172 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:22,172 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:22,172 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:22,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:22,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:22,172 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:22,172 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:22,172 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:22,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:22,272 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:22,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:22,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:22,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:22,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:22,276 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:22,276 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:22,276 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:22,276 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:22,276 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:22,276 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:22,277 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,277 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:22,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:22,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:22,277 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,277 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:22,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:22,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:22,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:22,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:22,278 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:22,278 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:22,278 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:22,278 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:22,278 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:22,279 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:22,279 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:22,279 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:22,279 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:22,279 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:22,279 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for aeecfead619437b2701aee4eba741c09: 2023-04-20 12:55:22,279 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:22,279 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:22,279 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:22,279 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,279 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:22,279 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:22,281 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:22,281 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:22,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:22,281 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:22,281 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:22,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:22,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:22,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:22,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:22,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:22,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f8dd454[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:22,283 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:22,283 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:22,283 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:22,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:22,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:22,283 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:22,283 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:22,283 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:22,284 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,284 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:22,284 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:22,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:22,284 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:22,284 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:22,285 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:22,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:22,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:22,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:22,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:22,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:22,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:22,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:22,297 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:22,297 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:22,297 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:22,297 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:22,297 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:22,298 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:22,299 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:22,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:22,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:22,299 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:22,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:22,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:22,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:22,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:22,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:22,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:22,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:22,300 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:22,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:22,301 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:22,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:22,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:22,307 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:22,307 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:22,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:22,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:22,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:22,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:22,308 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:22,308 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:22,307 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:22,307 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,308 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:22,308 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:22,308 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,308 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:22,308 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:22,308 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:22,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:22,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:22,309 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:22,408 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:22,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:22,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:22,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:22,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:22,411 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:22,411 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:22,411 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:22,411 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:22,412 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:22,412 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:22,412 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:22,412 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:22,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:22,413 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,413 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:22,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:22,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:22,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:22,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:22,413 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:22,414 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:22,414 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:22,414 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:22,414 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:22,414 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:22,414 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:22,414 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:22,414 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:22,414 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:22,415 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for aeecfead619437b2701aee4eba741c09: 2023-04-20 12:55:22,415 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:22,415 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:22,415 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:22,415 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,415 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:22,415 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:22,417 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,417 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:22,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:22,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:22,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:22,417 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:22,417 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:22,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:22,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:22,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:22,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:22,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cd8815f[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:22,418 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:22,418 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:22,419 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:22,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:22,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:22,419 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:22,419 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:22,419 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:22,420 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,420 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:22,420 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:22,420 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:22,420 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:22,420 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:22,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:22,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:22,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:22,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:22,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:22,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:22,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:22,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:22,425 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:22,425 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:22,426 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:22,426 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:22,426 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:22,432 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:22,432 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:22,432 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:22,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:22,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:22,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:22,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:22,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:22,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:22,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:22,433 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:22,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:22,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:22,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:22,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:22,434 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:22,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:22,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,436 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,436 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:22,436 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:22,437 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:22,437 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:22,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:22,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:22,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:22,437 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:22,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:22,437 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:22,437 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:22,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:22,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:22,437 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:22,437 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:22,437 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:22,437 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:22,538 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:22,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:22,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:22,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:22,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:22,541 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:22,541 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:22,541 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:22,541 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:22,541 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:22,541 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:22,542 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:22,542 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:22,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:22,542 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,542 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:22,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:22,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:22,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:22,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:22,543 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:22,554 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:22,554 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:22,554 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:22,554 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:22,554 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:22,554 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:22,554 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:22,554 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:22,555 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:22,555 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing aeecfead619437b2701aee4eba741c09 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-20 12:55:22,560 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=A 2023-04-20 12:55:22,560 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:22,560 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=B 2023-04-20 12:55:22,560 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:22,560 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=C 2023-04-20 12:55:22,560 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:22,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on aeecfead619437b2701aee4eba741c09 2023-04-20 12:55:22,579 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:22,649 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681995322635, max timestamp=1681995322648Num uniques 859; 2023-04-20 12:55:22,650 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681995322635, max timestamp=1681995322648Num uniques 859; 2023-04-20 12:55:22,650 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-20 12:55:22,650 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-20 12:55:22,649 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681995322635, max timestamp=1681995322648Num uniques 859; 2023-04-20 12:55:22,650 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-20 12:55:22,708 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1681995322686, max timestamp=1681995322706Num uniques 1118; 2023-04-20 12:55:22,708 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=14558, suffix heap size=144240,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=318 2023-04-20 12:55:22,709 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1681995322686, max timestamp=1681995322706Num uniques 1118; 2023-04-20 12:55:22,709 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=14558, suffix heap size=144240,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=318 2023-04-20 12:55:22,710 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1681995322686, max timestamp=1681995322706Num uniques 1118; 2023-04-20 12:55:22,710 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=14558, suffix heap size=144240,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=318 2023-04-20 12:55:22,742 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=427, cellSize=19552, totalHeapSize=44648, min timestamp=1681995322715, max timestamp=1681995322739Num uniques 1177; 2023-04-20 12:55:22,742 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=19552, suffix heap size=150376,new segment heap size=44648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=427 2023-04-20 12:55:22,742 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=427, cellSize=19552, totalHeapSize=44648, min timestamp=1681995322715, max timestamp=1681995322739Num uniques 1177; 2023-04-20 12:55:22,742 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=427, cellSize=19552, totalHeapSize=44648, min timestamp=1681995322715, max timestamp=1681995322739Num uniques 1177; 2023-04-20 12:55:22,742 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=19552, suffix heap size=150376,new segment heap size=44648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=427 2023-04-20 12:55:22,742 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=19552, suffix heap size=150376,new segment heap size=44648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=427 2023-04-20 12:55:22,779 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:22,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35842 deadline: 1681995382770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,784 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:22,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35886 deadline: 1681995382772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,785 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:22,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35900 deadline: 1681995382772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,785 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:22,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35848 deadline: 1681995382773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,785 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:22,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35892 deadline: 1681995382773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,889 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:22,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35842 deadline: 1681995382881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,894 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:22,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35886 deadline: 1681995382885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,894 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:22,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35900 deadline: 1681995382886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,895 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:22,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35848 deadline: 1681995382886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:22,895 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:22,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35892 deadline: 1681995382886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:23,012 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304206ad8a607aeff4c50955c313e1f5dd5b2 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304206ad8a607aeff4c50955c313e1f5dd5b2 2023-04-20 12:55:23,018 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=192, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/A/3a1b8587a3614360b7b4b7becf759b65 2023-04-20 12:55:23,093 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:23,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35842 deadline: 1681995383091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:23,099 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:23,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35886 deadline: 1681995383095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:23,099 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:23,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35892 deadline: 1681995383096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:23,099 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:23,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35900 deadline: 1681995383096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:23,099 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:23,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35848 deadline: 1681995383097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:23,402 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:23,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35842 deadline: 1681995383396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:23,403 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:23,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35892 deadline: 1681995383400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:23,403 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:23,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35900 deadline: 1681995383400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:23,403 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:23,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35886 deadline: 1681995383400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:23,405 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:23,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35848 deadline: 1681995383402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:23,432 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=192 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/B/4420dd4bd54a4f7f834e4680f39dfa4a 2023-04-20 12:55:23,440 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=192 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/C/a8d86a2e24f747a8bd2d01f968016666 2023-04-20 12:55:23,448 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/A/3a1b8587a3614360b7b4b7becf759b65 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/3a1b8587a3614360b7b4b7becf759b65 2023-04-20 12:55:23,451 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/3a1b8587a3614360b7b4b7becf759b65, entries=200, sequenceid=192, filesize=32.1 K 2023-04-20 12:55:23,452 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/B/4420dd4bd54a4f7f834e4680f39dfa4a as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/4420dd4bd54a4f7f834e4680f39dfa4a 2023-04-20 12:55:23,454 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/4420dd4bd54a4f7f834e4680f39dfa4a, entries=100, sequenceid=192, filesize=9.4 K 2023-04-20 12:55:23,455 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/C/a8d86a2e24f747a8bd2d01f968016666 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/a8d86a2e24f747a8bd2d01f968016666 2023-04-20 12:55:23,458 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/a8d86a2e24f747a8bd2d01f968016666, entries=100, sequenceid=192, filesize=9.4 K 2023-04-20 12:55:23,459 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=161.02 KB/164880 for aeecfead619437b2701aee4eba741c09 in 904ms, sequenceid=192, compaction requested=true 2023-04-20 12:55:23,459 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for aeecfead619437b2701aee4eba741c09: 2023-04-20 12:55:23,459 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:23,459 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:23,459 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:23,459 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:23,459 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:23,459 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:23,460 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:23,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:23,460 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:23,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:23,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:23,460 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:23,461 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:23,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:23,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:23,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:23,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:23,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:23,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:23,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ccbd6de[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:23,462 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:23,462 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:23,462 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:23,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:23,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:23,462 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:23,462 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:23,462 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:23,462 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:23,462 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:23,463 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:23,463 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:23,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:23,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:23,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:23,463 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:23,463 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:23,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:23,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:23,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:23,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:23,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:23,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:23,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:23,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:23,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:23,466 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:23,466 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:23,466 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:23,466 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:23,466 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:23,467 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:23,467 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:23,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:23,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:23,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:23,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:23,467 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:23,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:23,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:23,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:23,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:23,468 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:23,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:23,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:23,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:23,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:23,468 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:23,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:23,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:23,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:23,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:23,470 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:23,470 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:23,470 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:23,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:23,470 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:23,470 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:23,470 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:23,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:23,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:23,470 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:23,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:23,470 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:23,470 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:23,470 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:23,470 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:23,471 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:23,471 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:23,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:23,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:23,571 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:23,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:23,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:23,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:23,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:23,573 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:23,573 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:23,573 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:23,573 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:23,574 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:23,574 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:23,574 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:23,574 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:23,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:23,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:23,575 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:23,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:23,575 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:23,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:23,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:23,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:23,576 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:23,576 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:23,576 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:23,576 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:23,576 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:23,576 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:23,576 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:23,576 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:23,576 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:23,576 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:23,576 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing aeecfead619437b2701aee4eba741c09 3/3 column families, dataSize=161.02 KB heapSize=439.55 KB 2023-04-20 12:55:23,577 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=A 2023-04-20 12:55:23,577 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:55:23,577 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=B 2023-04-20 12:55:23,577 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:55:23,577 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=C 2023-04-20 12:55:23,577 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:55:23,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on aeecfead619437b2701aee4eba741c09 2023-04-20 12:55:23,910 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:23,912 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:23,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35886 deadline: 1681995383908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:23,916 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:23,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35900 deadline: 1681995383911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:23,916 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:23,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35842 deadline: 1681995383911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:23,918 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:23,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35848 deadline: 1681995383911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:23,918 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:23,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35892 deadline: 1681995383912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:23,999 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304202166760463ee48c7970d872b62952f9a to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304202166760463ee48c7970d872b62952f9a 2023-04-20 12:55:24,020 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:24,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35900 deadline: 1681995384016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:24,020 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:24,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35842 deadline: 1681995384017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:24,022 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:24,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35848 deadline: 1681995384018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:24,023 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:24,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35892 deadline: 1681995384019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:24,224 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:24,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35900 deadline: 1681995384221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:24,224 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:24,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35842 deadline: 1681995384221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:24,224 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:24,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35892 deadline: 1681995384223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:24,229 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:24,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35848 deadline: 1681995384224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:24,404 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=262, memsize=53.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/A/04630b82119948859e7dfe3a98cef1b7 2023-04-20 12:55:24,525 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:24,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35900 deadline: 1681995384525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:24,532 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:24,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35842 deadline: 1681995384527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:24,533 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:24,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35892 deadline: 1681995384527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:24,537 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:24,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35848 deadline: 1681995384532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:24,813 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=262 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/B/4f3649c9ede34589a1dedcd58521730b 2023-04-20 12:55:24,822 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=262 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/C/e0aa0f8b65d34b6da6f9dadaa3cca114 2023-04-20 12:55:24,827 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/A/04630b82119948859e7dfe3a98cef1b7 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/04630b82119948859e7dfe3a98cef1b7 2023-04-20 12:55:24,831 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/04630b82119948859e7dfe3a98cef1b7, entries=150, sequenceid=262, filesize=25.4 K 2023-04-20 12:55:24,831 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/B/4f3649c9ede34589a1dedcd58521730b as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/4f3649c9ede34589a1dedcd58521730b 2023-04-20 12:55:24,836 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/4f3649c9ede34589a1dedcd58521730b, entries=150, sequenceid=262, filesize=11.9 K 2023-04-20 12:55:24,836 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/C/e0aa0f8b65d34b6da6f9dadaa3cca114 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/e0aa0f8b65d34b6da6f9dadaa3cca114 2023-04-20 12:55:24,840 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/e0aa0f8b65d34b6da6f9dadaa3cca114, entries=150, sequenceid=262, filesize=11.9 K 2023-04-20 12:55:24,841 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~439.50 KB/450048, currentSize=26.84 KB/27480 for aeecfead619437b2701aee4eba741c09 in 1265ms, sequenceid=262, compaction requested=true 2023-04-20 12:55:24,841 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for aeecfead619437b2701aee4eba741c09: 2023-04-20 12:55:24,841 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:24,841 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:24,841 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:24,841 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:24,841 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:24,841 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:24,842 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:24,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:24,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:24,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:24,842 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:24,843 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:24,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:24,843 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:24,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:24,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:24,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:24,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:24,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:24,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d82c39c[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:24,844 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:24,844 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:24,845 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:24,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:24,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:24,845 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:24,845 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:24,845 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:24,845 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:24,845 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:24,846 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:24,850 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:24,851 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:24,851 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:24,851 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:24,851 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:24,851 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:24,851 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:24,851 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:24,853 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:24,853 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:24,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:24,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:24,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:24,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:24,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:24,853 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:24,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:24,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:24,853 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:24,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:24,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:24,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:24,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:24,854 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:24,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:24,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:24,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:24,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:24,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:24,856 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:24,856 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:24,856 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:24,856 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:24,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:24,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:24,856 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:24,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:24,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:24,856 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:24,856 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:24,857 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:24,857 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:24,856 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:24,857 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:24,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:24,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:24,857 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:24,857 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:24,957 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:24,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:24,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:24,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:24,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:24,974 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:24,975 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:24,975 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:24,975 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:24,975 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:24,975 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:24,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on aeecfead619437b2701aee4eba741c09 2023-04-20 12:55:24,975 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing aeecfead619437b2701aee4eba741c09 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-20 12:55:24,975 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=A 2023-04-20 12:55:24,976 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:24,976 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=B 2023-04-20 12:55:24,976 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:24,976 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=C 2023-04-20 12:55:24,976 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:24,976 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:24,976 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:24,976 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:24,976 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:24,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:24,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:24,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:24,977 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:24,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:24,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:24,978 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:24,988 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:24,988 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:24,988 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:24,988 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:24,988 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:24,989 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:24,989 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:24,989 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:24,990 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:24,990 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:24,990 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:24,990 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:24,990 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:24,990 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:24,998 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:24,998 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:24,999 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:24,999 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:24,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:24,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:24,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:24,999 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:24,999 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:24,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:25,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:25,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:25,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:25,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:25,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cec870c[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:25,001 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:25,001 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,002 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,002 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:25,002 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:25,002 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,003 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:25,002 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:25,005 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:25,005 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:25,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:25,005 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:25,005 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:25,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:25,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:25,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:25,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:25,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:25,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:25,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:25,007 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:25,007 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:25,007 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:25,007 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:25,007 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:25,009 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,009 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:25,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,009 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:25,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:25,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:25,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:25,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:25,010 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:25,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:25,011 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:25,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:25,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:25,013 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,013 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:25,013 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:25,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:25,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:25,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:25,013 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:25,013 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:25,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:25,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:25,013 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:25,013 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:25,013 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:25,013 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,013 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,013 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,013 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:25,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:25,113 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681995325097, max timestamp=1681995325111Num uniques 859; 2023-04-20 12:55:25,113 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681995325097, max timestamp=1681995325111Num uniques 859; 2023-04-20 12:55:25,113 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-20 12:55:25,113 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681995325097, max timestamp=1681995325111Num uniques 859; 2023-04-20 12:55:25,113 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-20 12:55:25,113 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-20 12:55:25,113 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:25,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:25,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:25,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:25,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:25,117 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:25,117 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:25,117 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:25,117 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:25,117 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,117 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:25,118 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,118 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:25,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:25,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:25,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:25,118 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,118 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:25,119 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:25,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:25,119 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:25,119 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:25,119 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:25,119 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:25,119 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:25,119 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:25,121 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:25,121 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:25,121 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:25,121 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:25,121 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:25,121 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:25,121 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:25,121 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:25,121 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,121 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:25,122 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:25,132 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,132 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:25,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:25,132 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,132 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:25,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:25,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:25,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:25,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:25,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:25,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4863bb7[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:25,134 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:25,134 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,136 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,136 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:25,136 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:25,136 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:25,137 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:25,137 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:25,137 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:25,138 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,138 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:25,139 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:25,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:25,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:25,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:25,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:25,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:25,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:25,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:25,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:25,142 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:25,142 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:25,142 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:25,142 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:25,142 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:25,143 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,143 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:25,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:25,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:25,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,145 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:25,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:25,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:25,145 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:25,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:25,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:25,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,147 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:25,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:25,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,149 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:25,149 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:25,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:25,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:25,149 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,149 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:25,149 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:25,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:25,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:25,149 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:25,149 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:25,149 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,149 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,149 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:25,150 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:25,150 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:25,154 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:25,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35900 deadline: 1681995385140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,162 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:25,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35848 deadline: 1681995385151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,163 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:25,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35892 deadline: 1681995385152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,163 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:25,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35886 deadline: 1681995385153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,163 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:25,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35842 deadline: 1681995385154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,250 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:25,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:25,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:25,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:25,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:25,253 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:25,253 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:25,253 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:25,253 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:25,254 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,254 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:25,254 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:25,254 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:25,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:25,255 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:25,255 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:25,255 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:25,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:25,255 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:25,261 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:25,262 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:25,262 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:25,262 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:25,262 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:25,262 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:25,262 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:25,262 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:25,262 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:25,262 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:25,262 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:25,262 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:25,262 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:25,262 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,262 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:25,262 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:25,262 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:25,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35900 deadline: 1681995385255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,264 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:25,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35848 deadline: 1681995385263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,269 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,269 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:25,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:25,270 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,270 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:25,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:25,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:25,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:25,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:25,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:25,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@265960df[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:25,273 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:25,273 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,274 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,274 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:25,274 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:25,274 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:25,274 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,274 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:25,274 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:25,275 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,275 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:25,275 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:25,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:25,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:25,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:25,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:25,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:25,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:25,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:25,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:25,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:25,276 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:25,277 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:25,277 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:25,277 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:25,277 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:25,277 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:25,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35892 deadline: 1681995385264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,277 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:25,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35886 deadline: 1681995385264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,277 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:25,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35842 deadline: 1681995385264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,279 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,279 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:25,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:25,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:25,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,280 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,282 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:25,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:25,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:25,293 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:25,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:25,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:25,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:25,296 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:25,296 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:25,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:25,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:25,296 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,296 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:25,296 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:25,296 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,296 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,296 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,296 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:25,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:25,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:25,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:25,296 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:25,297 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:25,297 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:25,297 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:25,397 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:25,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:25,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:25,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:25,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:25,414 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:25,414 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:25,414 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:25,414 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:25,415 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,415 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:25,417 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,418 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:25,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:25,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:25,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:25,418 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:25,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:25,419 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:25,419 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,419 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:25,423 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:25,423 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:25,423 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:25,423 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:25,423 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:25,424 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:25,424 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:25,424 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:25,424 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:25,424 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:25,424 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:25,424 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:25,424 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:25,424 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,424 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:25,424 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:25,428 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,428 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:25,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:25,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:25,428 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,429 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:25,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:25,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:25,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:25,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:25,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d2cc9ab[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:25,430 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:25,430 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,436 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,436 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:25,436 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:25,436 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:25,437 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,437 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:25,438 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,438 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:25,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:25,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:25,438 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:25,438 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:25,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:25,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:25,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:25,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:25,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:25,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:25,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:25,440 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:25,441 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:25,441 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:25,441 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:25,441 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:25,443 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,443 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:25,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:25,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:25,443 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:25,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:25,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:25,445 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:25,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:25,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:25,448 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:25,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:25,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,450 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,450 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:25,450 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:25,450 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:25,450 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:25,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:25,450 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:25,450 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:25,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:25,450 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:25,450 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,450 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:25,450 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:25,450 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:25,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:25,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:25,462 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304202ecb8d5d8a194852b1573ae28705aeee to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304202ecb8d5d8a194852b1573ae28705aeee 2023-04-20 12:55:25,475 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:25,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35900 deadline: 1681995385465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,475 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:25,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35848 deadline: 1681995385466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,490 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:25,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35842 deadline: 1681995385479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,491 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:25,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35892 deadline: 1681995385479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,491 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:25,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35886 deadline: 1681995385479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,550 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:25,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:25,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:25,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:25,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:25,556 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:25,556 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:25,556 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:25,556 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:25,557 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,557 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:25,558 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:25,558 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:25,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:25,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:25,559 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,559 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:25,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:25,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:25,559 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:25,562 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:25,562 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:25,562 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:25,562 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:25,562 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:25,571 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:25,571 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:25,582 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:25,583 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:25,583 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:25,583 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:25,583 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:25,583 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:25,583 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,583 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:25,583 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:25,585 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,585 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:25,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:25,585 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,585 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:25,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:25,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:25,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:25,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:25,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:25,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1520f3e5[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:25,587 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:25,587 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,588 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,588 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:25,588 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,588 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:25,589 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:25,589 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:25,589 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:25,590 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:25,589 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,590 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:25,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:25,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:25,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:25,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:25,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:25,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:25,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:25,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:25,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:25,592 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:25,592 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:25,592 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:25,592 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:25,592 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:25,593 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,593 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:25,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,593 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:25,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:25,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:25,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:25,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:25,594 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:25,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:25,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:25,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,596 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:25,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:25,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,597 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:25,597 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:25,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:25,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:25,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:25,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:25,598 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:25,598 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,598 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:25,598 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:25,598 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,598 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,598 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,598 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:25,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:25,599 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:25,599 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:25,699 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:25,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:25,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:25,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:25,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:25,702 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:25,702 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:25,702 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:25,702 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:25,702 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,702 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:25,703 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:25,703 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:25,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:25,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:25,703 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,703 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:25,704 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:25,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:25,704 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:25,704 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:25,704 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:25,704 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:25,704 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:25,704 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:25,706 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:25,706 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:25,706 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:25,706 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:25,706 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:25,706 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:25,706 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:25,706 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:25,706 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,706 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:25,706 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:25,707 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,707 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:25,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:25,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:25,708 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,708 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:25,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:25,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:25,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:25,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:25,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@204e3975[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:25,711 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:25,711 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,720 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,720 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:25,720 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:25,720 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:25,721 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,721 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:25,723 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,723 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:25,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:25,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:25,724 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:25,724 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:25,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:25,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:25,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:25,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:25,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:25,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:25,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:25,727 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:25,727 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:25,727 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:25,727 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:25,727 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:25,732 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,732 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:25,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:25,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,732 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:25,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,732 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:25,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:25,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:25,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:25,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:25,733 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:25,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:25,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:25,736 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,736 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:25,736 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:25,736 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:25,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:25,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:25,736 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,736 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,736 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,736 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,736 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:25,736 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:25,736 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:25,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:25,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:25,736 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:25,736 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:25,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:25,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:25,784 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:25,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35900 deadline: 1681995385777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,785 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:25,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35848 deadline: 1681995385778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,799 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:25,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35842 deadline: 1681995385793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,799 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:25,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35886 deadline: 1681995385793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,799 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:25,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35892 deadline: 1681995385794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,837 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:25,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:25,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:25,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:25,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:25,839 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:25,839 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:25,840 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:25,840 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:25,840 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,840 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:25,840 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,841 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:25,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:25,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:25,841 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,841 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:25,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:25,841 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:25,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:25,842 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:25,842 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:25,842 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:25,842 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:25,842 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:25,842 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:25,842 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:25,842 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:25,842 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:25,842 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:25,842 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:25,842 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:25,842 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:25,843 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,843 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:25,843 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:25,843 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,843 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:25,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:25,844 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:25,844 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:25,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:25,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:25,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:25,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:25,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d7cea3b[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:25,845 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:25,845 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,845 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,846 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:25,846 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:25,846 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:25,846 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,846 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:25,846 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,846 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:25,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:25,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:25,846 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:25,847 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:25,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:25,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:25,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:25,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:25,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:25,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:25,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:25,849 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:25,849 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:25,849 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:25,849 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:25,849 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:25,850 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,850 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,850 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:25,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:25,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:25,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:25,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:25,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:25,851 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:25,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:25,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:25,852 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:25,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:25,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,854 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,854 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:25,854 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:25,854 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:25,854 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:25,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:25,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:25,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:25,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:25,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:25,854 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,854 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:25,855 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,855 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:25,855 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:25,855 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:25,855 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:25,906 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=272, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/A/1ff2247e942a49f58c43f4676e604e56 2023-04-20 12:55:25,915 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=272 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/B/e94fd2b13b5f4eb9a6245e28028a9222 2023-04-20 12:55:25,955 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:25,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:25,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:25,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:25,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:25,958 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:25,958 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:25,958 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:25,958 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:25,958 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,958 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:25,959 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,959 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:25,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:25,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:25,959 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,959 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:25,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:25,960 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:25,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:25,960 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:25,960 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:25,960 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:25,960 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:25,960 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:25,960 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:25,960 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:25,960 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:25,960 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:25,960 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:25,960 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:25,960 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:25,961 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:25,961 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:25,961 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,961 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:25,961 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:25,961 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,961 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:25,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:25,962 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,962 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:25,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:25,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:25,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:25,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:25,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:25,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@663603ae[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:25,963 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:25,963 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,963 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,963 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:25,963 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:25,963 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:25,964 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,964 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:25,964 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:25,964 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:25,964 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:25,965 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:25,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:25,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:25,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:25,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:25,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:25,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:25,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:25,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:25,967 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:25,967 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:25,967 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:25,967 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:25,967 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:25,968 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:25,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:25,968 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,968 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:25,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:25,968 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:25,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:25,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:25,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:25,972 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:25,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:25,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:25,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:25,978 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,978 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:25,978 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:25,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:25,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:25,978 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:25,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:25,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:25,978 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:25,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:25,978 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:25,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:25,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:25,978 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:25,978 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:25,978 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:25,978 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,978 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:25,978 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:26,078 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:26,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:26,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:26,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:26,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:26,081 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:26,081 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:26,081 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:26,081 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:26,081 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:26,081 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:26,082 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:26,082 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:26,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:26,082 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,082 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:26,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:26,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:26,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:26,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:26,083 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:26,083 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:26,083 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:26,083 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:26,084 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:26,084 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:26,084 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:26,084 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:26,084 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:26,084 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:26,084 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:26,084 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:26,084 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:26,084 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:26,084 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,084 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:26,084 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:26,085 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,085 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:26,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:26,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:26,085 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:26,085 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:26,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:26,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:26,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:26,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:26,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:26,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@741b7b56[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:26,086 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:26,086 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:26,087 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:26,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:26,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:26,087 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:26,087 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:26,087 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:26,087 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,087 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:26,087 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,087 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:26,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:26,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:26,087 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:26,088 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:26,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:26,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:26,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:26,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:26,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:26,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:26,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:26,089 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:26,089 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:26,089 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:26,089 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:26,089 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:26,090 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:26,090 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:26,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:26,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:26,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:26,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:26,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:26,090 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:26,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:26,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:26,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:26,091 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:26,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:26,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:26,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:26,092 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:26,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:26,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,093 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,093 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:26,093 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:26,093 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:26,093 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:26,093 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:26,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:26,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:26,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:26,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:26,093 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,093 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:26,093 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:26,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:26,093 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:26,093 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:26,093 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:26,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:26,094 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:26,194 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:26,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:26,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:26,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:26,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:26,196 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:26,196 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:26,196 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:26,196 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:26,197 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:26,197 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:26,197 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:26,197 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:26,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:26,197 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,197 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:26,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:26,198 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:26,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:26,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:26,198 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:26,198 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:26,198 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:26,198 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:26,198 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:26,198 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:26,199 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:26,199 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:26,199 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:26,199 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:26,199 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:26,199 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:26,199 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:26,199 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:26,199 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,199 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:26,199 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:26,200 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,200 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:26,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:26,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:26,200 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:26,200 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:26,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:26,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:26,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:26,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:26,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:26,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@254f6131[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:26,201 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:26,201 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:26,202 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:26,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:26,202 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,202 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:26,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:26,202 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:26,202 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:26,202 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:26,203 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,203 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:26,203 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:26,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:26,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:26,203 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:26,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:26,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:26,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:26,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:26,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:26,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:26,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:26,204 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:26,204 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:26,204 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:26,204 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:26,204 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:26,205 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:26,205 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:26,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:26,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:26,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:26,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:26,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:26,205 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:26,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:26,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:26,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:26,206 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:26,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:26,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:26,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:26,207 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:26,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:26,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,208 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,209 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:26,209 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:26,209 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:26,209 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:26,209 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:26,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:26,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:26,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:26,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:26,209 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:26,209 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,209 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:26,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:26,209 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:26,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:26,209 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:26,209 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:26,209 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:26,292 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:26,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35900 deadline: 1681995386285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,292 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:26,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35848 deadline: 1681995386288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,307 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:26,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35886 deadline: 1681995386300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,307 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:26,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35892 deadline: 1681995386300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,308 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:26,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35842 deadline: 1681995386301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,309 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:26,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:26,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:26,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:26,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:26,312 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:26,312 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:26,312 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:26,312 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:26,312 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:26,312 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:26,313 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,313 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:26,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:26,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:26,313 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,313 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:26,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:26,313 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:26,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:26,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:26,313 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:26,313 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:26,313 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:26,314 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:26,314 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:26,314 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:26,314 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:26,314 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:26,314 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:26,314 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:26,314 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:26,314 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:26,314 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:26,314 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:26,315 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,315 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:26,315 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:26,315 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,315 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:26,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:26,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:26,316 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:26,316 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:26,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:26,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:26,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:26,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:26,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:26,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f486a91[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:26,319 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:26,319 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:26,321 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:26,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:26,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:26,321 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:26,321 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:26,321 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:26,322 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,322 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:26,322 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,322 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:26,322 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:26,322 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:26,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:26,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:26,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:26,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:26,323 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=272 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/C/d470330bb8ac4347873843d2457be056 2023-04-20 12:55:26,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:26,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:26,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:26,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:26,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:26,325 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:26,325 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:26,325 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:26,325 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:26,325 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:26,327 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:26,327 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:26,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:26,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:26,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:26,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:26,327 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:26,328 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,329 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:26,329 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:26,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:26,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:26,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:26,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:26,330 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:26,330 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:26,330 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,330 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:26,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:26,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:26,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:26,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:26,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:26,330 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:26,330 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:26,332 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,332 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:26,332 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:26,332 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:26,332 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:26,332 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/A/1ff2247e942a49f58c43f4676e604e56 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/1ff2247e942a49f58c43f4676e604e56 2023-04-20 12:55:26,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:26,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:26,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:26,335 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:26,338 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/1ff2247e942a49f58c43f4676e604e56, entries=150, sequenceid=272, filesize=25.4 K 2023-04-20 12:55:26,338 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/B/e94fd2b13b5f4eb9a6245e28028a9222 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/e94fd2b13b5f4eb9a6245e28028a9222 2023-04-20 12:55:26,341 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/e94fd2b13b5f4eb9a6245e28028a9222, entries=100, sequenceid=272, filesize=9.5 K 2023-04-20 12:55:26,342 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/C/d470330bb8ac4347873843d2457be056 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/d470330bb8ac4347873843d2457be056 2023-04-20 12:55:26,348 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/d470330bb8ac4347873843d2457be056, entries=100, sequenceid=272, filesize=9.5 K 2023-04-20 12:55:26,349 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for aeecfead619437b2701aee4eba741c09 in 1374ms, sequenceid=272, compaction requested=true 2023-04-20 12:55:26,349 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for aeecfead619437b2701aee4eba741c09: 2023-04-20 12:55:26,349 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:55:26,349 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-20 12:55:26,349 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-20 12:55:26,349 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:55:26,349 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:55:26,351 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 68020 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-20 12:55:26,351 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): aeecfead619437b2701aee4eba741c09/B is initiating minor compaction (all files) 2023-04-20 12:55:26,352 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of aeecfead619437b2701aee4eba741c09/B in TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:26,352 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/70eb4f2936b647dea43220522ef33d65, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/ec6d7cd665c749acb0b0b4b324a52f5a, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/5b7415cb7795431282c6834f9b0b741e, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/4420dd4bd54a4f7f834e4680f39dfa4a, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/4f3649c9ede34589a1dedcd58521730b, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/e94fd2b13b5f4eb9a6245e28028a9222] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp, totalSize=66.4 K 2023-04-20 12:55:26,352 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 163000 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-20 12:55:26,352 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): aeecfead619437b2701aee4eba741c09/A is initiating minor compaction (all files) 2023-04-20 12:55:26,352 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of aeecfead619437b2701aee4eba741c09/A in TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:26,352 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/d1bdd99960734e19a03f7f0b2a311120, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/6517274f9a5d4c37a48ac0c3c50042d4, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/0261f2f8bf934ef49e53cd0b795110ab, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/3a1b8587a3614360b7b4b7becf759b65, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/04630b82119948859e7dfe3a98cef1b7, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/1ff2247e942a49f58c43f4676e604e56] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp, totalSize=159.2 K 2023-04-20 12:55:26,352 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 70eb4f2936b647dea43220522ef33d65, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1681995318553 2023-04-20 12:55:26,352 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting d1bdd99960734e19a03f7f0b2a311120, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1681995318553 2023-04-20 12:55:26,353 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting ec6d7cd665c749acb0b0b4b324a52f5a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=166, earliestPutTs=1681995319195 2023-04-20 12:55:26,353 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 6517274f9a5d4c37a48ac0c3c50042d4, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=166, earliestPutTs=1681995319195 2023-04-20 12:55:26,353 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 5b7415cb7795431282c6834f9b0b741e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1681995320340 2023-04-20 12:55:26,353 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 0261f2f8bf934ef49e53cd0b795110ab, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1681995320340 2023-04-20 12:55:26,354 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 4420dd4bd54a4f7f834e4680f39dfa4a, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1681995322523 2023-04-20 12:55:26,354 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 3a1b8587a3614360b7b4b7becf759b65, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1681995322521 2023-04-20 12:55:26,354 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 4f3649c9ede34589a1dedcd58521730b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1681995322769 2023-04-20 12:55:26,354 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 04630b82119948859e7dfe3a98cef1b7, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1681995322769 2023-04-20 12:55:26,354 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting e94fd2b13b5f4eb9a6245e28028a9222, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1681995324944 2023-04-20 12:55:26,355 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 1ff2247e942a49f58c43f4676e604e56, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1681995324944 2023-04-20 12:55:26,370 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): aeecfead619437b2701aee4eba741c09#B#compaction#349 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:55:26,377 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): aeecfead619437b2701aee4eba741c09#A#compaction#350 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:55:26,432 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:26,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:26,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:26,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:26,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:26,437 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:26,437 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:26,437 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:26,437 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:26,437 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:26,437 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:26,438 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:26,438 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:26,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:26,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:26,438 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:26,438 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:26,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:26,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:26,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:26,439 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:26,439 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:26,439 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:26,439 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:26,439 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:26,439 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:26,439 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:26,439 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:26,439 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:26,440 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:26,440 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing aeecfead619437b2701aee4eba741c09 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-20 12:55:26,440 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=A 2023-04-20 12:55:26,440 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:55:26,440 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=B 2023-04-20 12:55:26,440 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:55:26,440 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=C 2023-04-20 12:55:26,440 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:55:26,461 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420cc9fc10d69024da3bdc50672406562fc to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420cc9fc10d69024da3bdc50672406562fc 2023-04-20 12:55:26,467 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=307, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/A/eaa812e10aed4bc0b2d840b12e526078 2023-04-20 12:55:26,788 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/B/b246ac2492584758afef8d558c827473 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/b246ac2492584758afef8d558c827473 2023-04-20 12:55:26,793 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in aeecfead619437b2701aee4eba741c09/B of aeecfead619437b2701aee4eba741c09 into b246ac2492584758afef8d558c827473(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:55:26,793 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for aeecfead619437b2701aee4eba741c09: 2023-04-20 12:55:26,793 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09., storeName=aeecfead619437b2701aee4eba741c09/B, priority=10, startTime=1681995326349; duration=0sec 2023-04-20 12:55:26,794 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:55:26,794 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-20 12:55:26,795 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 68020 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-20 12:55:26,795 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): aeecfead619437b2701aee4eba741c09/C is initiating minor compaction (all files) 2023-04-20 12:55:26,795 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of aeecfead619437b2701aee4eba741c09/C in TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:26,795 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/c1f39d61f63b44068846f4978f1764c4, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/5d3f4c05b20d4528a9a0d17f013af497, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/8169a02f389446f7a783465051126740, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/a8d86a2e24f747a8bd2d01f968016666, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/e0aa0f8b65d34b6da6f9dadaa3cca114, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/d470330bb8ac4347873843d2457be056] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp, totalSize=66.4 K 2023-04-20 12:55:26,796 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting c1f39d61f63b44068846f4978f1764c4, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1681995318553 2023-04-20 12:55:26,796 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 5d3f4c05b20d4528a9a0d17f013af497, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=166, earliestPutTs=1681995319195 2023-04-20 12:55:26,796 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 8169a02f389446f7a783465051126740, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1681995320340 2023-04-20 12:55:26,796 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting a8d86a2e24f747a8bd2d01f968016666, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1681995322523 2023-04-20 12:55:26,796 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting e0aa0f8b65d34b6da6f9dadaa3cca114, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1681995322769 2023-04-20 12:55:26,797 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting d470330bb8ac4347873843d2457be056, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1681995324944 2023-04-20 12:55:26,804 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): aeecfead619437b2701aee4eba741c09#C#compaction#353 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:55:26,881 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=307 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/B/70920a1acff04c1abc16de623c78650c 2023-04-20 12:55:27,196 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/A/7918f59bb9b448968c29ded269a99b68 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/7918f59bb9b448968c29ded269a99b68 2023-04-20 12:55:27,200 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in aeecfead619437b2701aee4eba741c09/A of aeecfead619437b2701aee4eba741c09 into 7918f59bb9b448968c29ded269a99b68(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:55:27,200 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for aeecfead619437b2701aee4eba741c09: 2023-04-20 12:55:27,200 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09., storeName=aeecfead619437b2701aee4eba741c09/A, priority=10, startTime=1681995326349; duration=0sec 2023-04-20 12:55:27,200 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:55:27,211 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/C/d47d16e61b9444a09af0f6f7631f24c2 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/d47d16e61b9444a09af0f6f7631f24c2 2023-04-20 12:55:27,215 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in aeecfead619437b2701aee4eba741c09/C of aeecfead619437b2701aee4eba741c09 into d47d16e61b9444a09af0f6f7631f24c2(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:55:27,215 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for aeecfead619437b2701aee4eba741c09: 2023-04-20 12:55:27,215 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09., storeName=aeecfead619437b2701aee4eba741c09/C, priority=10, startTime=1681995326349; duration=0sec 2023-04-20 12:55:27,216 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:55:27,291 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=307 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/C/405807dd8f264aad8298596224a8ff33 2023-04-20 12:55:27,294 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/A/eaa812e10aed4bc0b2d840b12e526078 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/eaa812e10aed4bc0b2d840b12e526078 2023-04-20 12:55:27,294 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:27,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on aeecfead619437b2701aee4eba741c09 2023-04-20 12:55:27,297 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/eaa812e10aed4bc0b2d840b12e526078, entries=150, sequenceid=307, filesize=25.4 K 2023-04-20 12:55:27,297 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/B/70920a1acff04c1abc16de623c78650c as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/70920a1acff04c1abc16de623c78650c 2023-04-20 12:55:27,300 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/70920a1acff04c1abc16de623c78650c, entries=150, sequenceid=307, filesize=11.9 K 2023-04-20 12:55:27,301 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/C/405807dd8f264aad8298596224a8ff33 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/405807dd8f264aad8298596224a8ff33 2023-04-20 12:55:27,309 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/405807dd8f264aad8298596224a8ff33, entries=150, sequenceid=307, filesize=11.9 K 2023-04-20 12:55:27,309 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=6.71 KB/6870 for aeecfead619437b2701aee4eba741c09 in 869ms, sequenceid=307, compaction requested=false 2023-04-20 12:55:27,309 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for aeecfead619437b2701aee4eba741c09: 2023-04-20 12:55:27,310 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:27,310 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:27,310 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:27,310 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,310 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:27,310 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:27,311 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,311 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:27,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:27,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:27,311 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:27,311 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:27,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:27,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:27,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:27,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:27,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:27,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@593523a5[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:27,312 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:27,312 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:27,314 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:27,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:27,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:27,314 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:27,314 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:27,314 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,315 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:27,314 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:27,315 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,315 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:27,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:27,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:27,315 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:27,316 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:27,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:27,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:27,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:27,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:27,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:27,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:27,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:27,317 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:27,318 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:27,318 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:27,318 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:27,318 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:27,319 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,319 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:27,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:27,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:27,319 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,320 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,321 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,321 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:27,321 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,321 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:27,321 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:27,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:27,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:27,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:27,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:27,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:27,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:27,321 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:27,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:27,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:27,321 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:27,321 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:27,321 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,321 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:27,321 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:27,321 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:27,321 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:27,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:27,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:27,323 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:27,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on aeecfead619437b2701aee4eba741c09 2023-04-20 12:55:27,340 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing aeecfead619437b2701aee4eba741c09 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-20 12:55:27,340 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=A 2023-04-20 12:55:27,340 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:27,340 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=B 2023-04-20 12:55:27,340 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:27,340 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=C 2023-04-20 12:55:27,340 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:27,375 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681995327355, max timestamp=1681995327373Num uniques 859; 2023-04-20 12:55:27,375 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-20 12:55:27,375 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681995327355, max timestamp=1681995327373Num uniques 859; 2023-04-20 12:55:27,375 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-20 12:55:27,375 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681995327355, max timestamp=1681995327373Num uniques 859; 2023-04-20 12:55:27,375 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-20 12:55:27,400 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:27,400 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:27,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35886 deadline: 1681995387389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35892 deadline: 1681995387390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,402 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:27,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35842 deadline: 1681995387395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,402 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:27,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35900 deadline: 1681995387396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,402 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:27,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35848 deadline: 1681995387397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,421 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:27,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:27,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:27,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:27,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:27,424 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:27,424 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:27,424 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:27,424 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:27,424 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,424 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:27,425 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:27,425 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:27,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:27,425 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:27,425 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:27,426 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:27,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:27,426 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:27,426 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:27,426 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:27,426 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:27,426 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:27,426 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:27,426 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:27,426 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:27,426 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:27,427 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:27,427 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:27,427 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:27,427 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:27,427 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:27,427 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,427 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:27,427 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:27,428 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:27,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:27,428 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:27,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:27,428 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:27,428 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:27,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:27,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:27,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:27,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:27,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@765b321e[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:27,429 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:27,429 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:27,429 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:27,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:27,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:27,430 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,430 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:27,430 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:27,430 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:27,430 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:27,430 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,430 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:27,430 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:27,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:27,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:27,430 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:27,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:27,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:27,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:27,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:27,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:27,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:27,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:27,432 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:27,432 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:27,432 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:27,432 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:27,432 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:27,433 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,433 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:27,433 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:27,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:27,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:27,434 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:27,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:27,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:27,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:27,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:27,435 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:27,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:27,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,437 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,437 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:27,437 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:27,437 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:27,437 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:27,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:27,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:27,437 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,437 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:27,437 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:27,437 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:27,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:27,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:27,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:27,437 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:27,438 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:27,438 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:27,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:27,505 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:27,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35892 deadline: 1681995387501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,505 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:27,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35886 deadline: 1681995387501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,505 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:27,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35842 deadline: 1681995387503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,506 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:27,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35900 deadline: 1681995387503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,506 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:27,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35848 deadline: 1681995387503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,538 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:27,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:27,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:27,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:27,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:27,540 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:27,540 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:27,540 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:27,540 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:27,541 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,541 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:27,541 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:27,541 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:27,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:27,542 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,542 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:27,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:27,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:27,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:27,542 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:27,542 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:27,542 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:27,542 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:27,543 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:27,543 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:27,543 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:27,543 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:27,543 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:27,543 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:27,543 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:27,543 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:27,543 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:27,543 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:27,543 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,543 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:27,543 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:27,545 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:27,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:27,545 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:27,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:27,545 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:27,545 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:27,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:27,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:27,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:27,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:27,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28c3930d[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:27,546 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:27,546 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:27,547 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:27,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:27,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:27,547 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:27,547 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:27,547 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:27,547 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,547 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:27,548 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:27,548 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:27,548 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:27,548 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:27,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:27,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:27,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:27,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:27,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:27,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:27,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:27,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:27,550 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:27,550 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:27,550 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:27,550 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:27,550 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:27,551 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:27,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:27,551 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,551 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:27,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,551 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:27,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:27,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:27,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:27,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:27,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:27,552 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:27,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:27,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,555 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:27,555 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:27,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:27,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:27,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:27,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:27,556 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,556 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:27,556 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:27,556 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:27,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:27,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:27,556 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,556 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:27,556 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:27,556 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,556 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:27,557 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:27,557 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:27,657 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:27,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:27,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:27,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:27,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:27,660 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:27,660 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:27,660 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:27,660 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:27,660 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,660 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:27,661 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:27,661 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:27,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:27,661 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,661 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:27,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:27,662 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:27,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:27,662 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:27,662 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:27,662 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:27,663 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:27,663 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:27,663 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:27,663 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:27,663 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:27,663 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:27,663 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:27,663 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:27,663 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:27,663 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:27,663 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:27,663 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,663 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:27,663 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:27,664 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,664 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:27,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:27,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:27,665 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:27,665 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:27,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:27,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:27,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:27,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:27,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:27,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64137866[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:27,666 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:27,666 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:27,667 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:27,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:27,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:27,667 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:27,667 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,667 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:27,667 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:27,667 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:27,668 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:27,668 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,668 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:27,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:27,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:27,668 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:27,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:27,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:27,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:27,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:27,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:27,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:27,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:27,670 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:27,670 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:27,670 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:27,670 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:27,670 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:27,671 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,671 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:27,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,672 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:27,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:27,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:27,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:27,672 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:27,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:27,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:27,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:27,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:27,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:27,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,676 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,677 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,677 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:27,677 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:27,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:27,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:27,677 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:27,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:27,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:27,677 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:27,677 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:27,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:27,677 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,677 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:27,677 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:27,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:27,677 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:27,677 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,677 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:27,710 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:27,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35842 deadline: 1681995387706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,710 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:27,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35848 deadline: 1681995387707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,711 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:27,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35892 deadline: 1681995387707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,711 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:27,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35886 deadline: 1681995387707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,711 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:27,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35900 deadline: 1681995387707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,760 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304208ac144173e7f4b35b4125c93b6636a5d to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304208ac144173e7f4b35b4125c93b6636a5d 2023-04-20 12:55:27,777 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:27,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:27,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:27,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:27,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:27,780 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:27,780 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:27,780 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:27,780 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:27,781 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,781 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:27,781 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,781 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:27,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:27,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:27,781 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,782 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:27,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:27,782 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:27,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:27,782 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:27,782 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:27,782 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:27,782 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:27,782 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:27,782 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:27,783 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:27,783 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:27,783 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:27,783 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:27,783 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:27,783 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:27,783 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:27,783 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:27,783 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,783 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:27,783 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:27,786 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:27,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:27,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:27,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:27,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:27,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:27,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:27,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@144b574a[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:27,788 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:27,788 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:27,788 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:27,790 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:27,790 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:27,790 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:27,790 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:27,790 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:27,790 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,790 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:27,790 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:27,791 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:27,791 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:27,791 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:27,791 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:27,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:27,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:27,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:27,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:27,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:27,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:27,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:27,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:27,794 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:27,794 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:27,794 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:27,794 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:27,794 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:27,797 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:27,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:27,797 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,797 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:27,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,798 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:27,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:27,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:27,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:27,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:27,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:27,799 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:27,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:27,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,811 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,811 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:27,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:27,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:27,811 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:27,811 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:27,811 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:27,811 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:27,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:27,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:27,811 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:27,812 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:27,811 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,812 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:27,812 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:27,812 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,812 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:27,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:27,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:27,912 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:27,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:27,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:27,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:27,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:27,915 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:27,915 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:27,915 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:27,915 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:27,915 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,915 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:27,916 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,916 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:27,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:27,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:27,916 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,916 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:27,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:27,917 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:27,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:27,917 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:27,917 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:27,917 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:27,917 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:27,917 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:27,917 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:27,917 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:27,917 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:27,917 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:27,917 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:27,917 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:27,918 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:27,918 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:27,918 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:27,918 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,918 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:27,918 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:27,918 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,919 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:27,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:27,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:27,919 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:27,919 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:27,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:27,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:27,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:27,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:27,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:27,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b2f5c1f[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:27,920 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:27,920 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:27,921 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:27,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:27,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:27,921 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:27,921 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,921 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:27,921 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:27,922 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:27,922 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,922 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:27,922 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:27,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:27,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:27,922 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:27,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:27,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:27,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:27,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:27,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:27,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:27,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:27,924 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:27,924 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:27,924 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:27,924 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:27,924 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:27,925 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:27,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:27,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:27,925 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,925 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,925 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:27,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:27,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:27,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:27,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:27,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:27,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,926 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:27,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:27,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,927 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,927 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:27,927 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:27,927 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:27,927 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:27,928 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:27,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:27,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:27,928 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:27,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:27,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:27,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:27,928 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:27,928 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:27,928 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:27,928 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:27,928 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:27,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:27,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:28,015 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:28,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35848 deadline: 1681995388011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,016 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:28,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35842 deadline: 1681995388011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,016 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:28,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35892 deadline: 1681995388011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,016 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:28,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35886 deadline: 1681995388014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,016 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:28,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35900 deadline: 1681995388014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,028 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:28,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:28,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:28,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:28,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:28,031 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:28,031 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:28,031 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:28,031 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:28,031 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,031 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:28,032 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,032 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:28,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:28,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:28,032 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,032 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:28,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:28,032 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:28,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:28,032 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:28,033 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:28,033 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:28,033 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:28,033 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:28,033 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:28,033 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:28,033 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:28,033 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:28,033 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:28,033 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:28,033 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:28,033 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:28,033 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:28,033 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,033 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:28,033 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:28,034 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,034 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:28,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:28,034 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,034 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:28,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:28,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:28,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:28,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:28,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:28,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fc0eb8c[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:28,035 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:28,035 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,035 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,036 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:28,036 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,036 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:28,036 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:28,036 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:28,036 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:28,036 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,036 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:28,036 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:28,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:28,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:28,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:28,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:28,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:28,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:28,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:28,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:28,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:28,038 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:28,038 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:28,038 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:28,038 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:28,038 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:28,039 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,039 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,039 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:28,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:28,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:28,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:28,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:28,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,040 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:28,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:28,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:28,041 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,042 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,042 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:28,042 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:28,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:28,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:28,042 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:28,042 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:28,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:28,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:28,042 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:28,042 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:28,042 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:28,042 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,042 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:28,043 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:28,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:28,043 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,043 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,143 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:28,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:28,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:28,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:28,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:28,146 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:28,146 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:28,146 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:28,146 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:28,146 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,146 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:28,147 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:28,147 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:28,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:28,147 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,147 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:28,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:28,147 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:28,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:28,148 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:28,148 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:28,148 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:28,148 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:28,148 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:28,148 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:28,148 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:28,148 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:28,148 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:28,148 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:28,148 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:28,148 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:28,148 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:28,149 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:28,149 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,149 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:28,149 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:28,149 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,149 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:28,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:28,150 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,150 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:28,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:28,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:28,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:28,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:28,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:28,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70ef725a[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:28,150 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:28,151 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,152 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,152 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:28,153 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:28,153 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:28,153 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,153 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:28,153 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:28,153 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,153 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:28,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:28,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:28,153 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:28,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:28,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:28,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:28,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:28,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:28,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:28,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:28,155 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:28,155 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:28,155 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:28,155 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:28,155 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:28,156 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,156 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:28,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:28,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,156 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:28,156 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:28,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:28,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:28,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:28,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:28,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:28,157 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:28,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:28,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,159 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:28,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:28,159 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:28,159 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:28,159 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:28,159 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,159 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,159 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:28,159 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,159 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:28,159 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,159 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:28,159 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:28,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:28,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:28,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:28,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:28,164 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=320, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/A/59b48a0303a74a00803634790c142816 2023-04-20 12:55:28,259 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:28,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:28,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:28,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:28,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:28,263 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:28,263 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:28,263 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:28,263 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:28,264 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,264 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:28,271 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:28,271 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:28,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:28,271 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,271 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:28,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:28,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:28,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:28,272 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:28,273 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:28,273 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:28,273 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:28,273 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:28,273 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:28,274 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:28,274 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:28,274 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:28,275 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:28,275 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:28,275 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:28,275 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:28,275 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:28,275 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,275 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:28,275 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:28,278 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,278 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:28,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:28,278 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,278 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:28,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:28,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:28,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:28,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:28,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:28,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e509a0a[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:28,279 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:28,279 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,280 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,280 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:28,280 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,280 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:28,280 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:28,280 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:28,281 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,281 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:28,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:28,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:28,281 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:28,281 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:28,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:28,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:28,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:28,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:28,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:28,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:28,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:28,283 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:28,283 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:28,283 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:28,283 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:28,283 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:28,284 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,284 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:28,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,284 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:28,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:28,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:28,284 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:28,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:28,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:28,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:28,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:28,285 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:28,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:28,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,287 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,287 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:28,287 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:28,287 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,287 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,287 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:28,287 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:28,287 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:28,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:28,287 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:28,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:28,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:28,287 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:28,287 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:28,287 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:28,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:28,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:28,388 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:28,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:28,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:28,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:28,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:28,390 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:28,390 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:28,390 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:28,390 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:28,391 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,391 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:28,391 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:28,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:28,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:28,392 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:28,392 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,392 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:28,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:28,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:28,393 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:28,393 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:28,393 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:28,393 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:28,393 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:28,393 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:28,393 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:28,393 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:28,393 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:28,393 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:28,393 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:28,394 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:28,394 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:28,394 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:28,394 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,394 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:28,394 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:28,395 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,395 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:28,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:28,395 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,395 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:28,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:28,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:28,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:28,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:28,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:28,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f5ee67b[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:28,396 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:28,396 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,397 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,397 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:28,397 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,397 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:28,397 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:28,397 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:28,397 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,397 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:28,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:28,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:28,398 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:28,398 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:28,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:28,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:28,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:28,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:28,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:28,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:28,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:28,403 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:28,403 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:28,403 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:28,403 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:28,403 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:28,415 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:28,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:28,415 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,416 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:28,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:28,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:28,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:28,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:28,416 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:28,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:28,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,417 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:28,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:28,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,420 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,420 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:28,420 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:28,421 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,421 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,421 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,421 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,421 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:28,421 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:28,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:28,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:28,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:28,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:28,421 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:28,421 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:28,421 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:28,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:28,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:28,521 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:28,522 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:28,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35848 deadline: 1681995388518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:28,522 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:28,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:35892 deadline: 1681995388519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,522 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:28,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35886 deadline: 1681995388519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:28,524 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:28,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35900 deadline: 1681995388520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,524 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:28,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35842 deadline: 1681995388520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:28,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:28,525 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:28,525 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:28,525 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:28,525 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:28,525 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,525 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:28,526 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,526 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:28,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:28,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:28,526 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,526 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:28,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:28,527 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:28,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:28,527 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:28,527 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:28,527 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:28,527 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:28,527 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:28,527 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:28,527 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:28,527 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:28,527 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:28,528 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:28,528 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:28,528 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:28,528 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:28,528 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:28,528 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,528 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:28,528 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:28,529 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,529 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:28,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:28,529 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,529 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:28,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:28,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:28,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:28,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:28,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:28,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77cafe97[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:28,530 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:28,530 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,530 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,531 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:28,531 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:28,531 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,531 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:28,531 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:28,531 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,531 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:28,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:28,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:28,531 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:28,532 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:28,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:28,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:28,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:28,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:28,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:28,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:28,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:28,533 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:28,533 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:28,533 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:28,533 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:28,533 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:28,534 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,534 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,534 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:28,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:28,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:28,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:28,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:28,534 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:28,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:28,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:28,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:28,535 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:28,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:28,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,537 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,537 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:28,537 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:28,537 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:28,537 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:28,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:28,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:28,538 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,538 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,538 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,538 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:28,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:28,538 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:28,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:28,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:28,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:28,538 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:28,539 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:28,573 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/B/d294268ab3c243bd919ce29a5ecbeac3 2023-04-20 12:55:28,639 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:28,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:28,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:28,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:28,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:28,642 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:28,642 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:28,642 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:28,642 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:28,642 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,642 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:28,643 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:28,643 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:28,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:28,643 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,643 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:28,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:28,643 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:28,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:28,644 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:28,644 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:28,644 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:28,644 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:28,644 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:28,644 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:28,644 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:28,644 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:28,644 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:28,644 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:28,644 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:28,644 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:28,644 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:28,644 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:28,644 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,644 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:28,644 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:28,645 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,645 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:28,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:28,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:28,645 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,646 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:28,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:28,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:28,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:28,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:28,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c3c2f96[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:28,647 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:28,647 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,647 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,648 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:28,648 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,648 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:28,648 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:28,648 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:28,648 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,648 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:28,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:28,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:28,649 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:28,649 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:28,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:28,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:28,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:28,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:28,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:28,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:28,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:28,650 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:28,650 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:28,650 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:28,650 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:28,650 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:28,651 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,651 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:28,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:28,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,651 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:28,651 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:28,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:28,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:28,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:28,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:28,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:28,652 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:28,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:28,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,653 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,653 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:28,653 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:28,654 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:28,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:28,654 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:28,653 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:28,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:28,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:28,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:28,654 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:28,654 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,654 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:28,654 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,654 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:28,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:28,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:28,654 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,654 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,754 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:28,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:28,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:28,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:28,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:28,758 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:28,758 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:28,758 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:28,758 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:28,759 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,759 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:28,759 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,759 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:28,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:28,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:28,760 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:28,760 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:28,760 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:28,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:28,760 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:28,761 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:28,761 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:28,761 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:28,761 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:28,761 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:28,761 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:28,761 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:28,761 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:28,761 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:28,761 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:28,763 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:28,763 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:28,763 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,763 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:28,763 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:28,764 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,764 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:28,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:28,764 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:28,764 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:28,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:28,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:28,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:28,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:28,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46867ad1[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:28,765 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:28,765 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,765 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,766 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:28,766 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:28,766 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,766 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:28,766 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:28,766 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,766 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:28,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:28,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:28,766 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:28,767 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:28,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:28,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:28,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:28,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:28,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:28,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:28,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:28,770 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:28,770 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:28,770 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:28,770 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:28,770 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:28,772 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,772 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:28,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,772 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:28,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:28,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:28,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:28,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:28,773 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:28,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:28,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:28,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,775 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:28,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:28,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,779 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,779 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:28,779 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:28,779 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,779 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,779 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,779 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:28,779 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:28,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:28,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:28,779 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:28,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:28,779 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:28,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:28,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:28,779 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:28,779 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:28,880 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:28,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:28,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:28,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:28,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:28,882 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:28,882 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:28,882 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:28,882 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:28,883 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,883 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:28,883 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,883 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:28,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:28,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:28,884 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,884 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:28,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:28,884 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:28,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:28,884 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:28,885 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:28,885 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:28,885 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:28,885 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:28,885 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:28,885 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:28,885 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:28,885 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:28,885 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:28,885 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:28,885 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:28,885 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:28,885 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:28,885 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,885 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:28,885 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:28,886 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,886 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:28,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:28,886 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,886 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:28,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:28,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:28,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:28,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:28,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:28,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e633144[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:28,887 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:28,887 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,888 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,888 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:28,888 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:28,888 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:28,888 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,888 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:28,888 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,888 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:28,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:28,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:28,889 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:28,889 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:28,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:28,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:28,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:28,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:28,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:28,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:28,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:28,890 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:28,891 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:28,891 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:28,891 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:28,891 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:28,892 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,892 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:28,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:28,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,892 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:28,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:28,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:28,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:28,892 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:28,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:28,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:28,894 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:28,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:28,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,895 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,895 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:28,895 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:28,895 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:28,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:28,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:28,895 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:28,895 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:28,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:28,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:28,895 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:28,895 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:28,896 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:28,896 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:28,895 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:28,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:28,896 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:28,896 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:28,982 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/C/4a9436b4ec074a6a95d397965003e8cb 2023-04-20 12:55:28,988 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/A/59b48a0303a74a00803634790c142816 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/59b48a0303a74a00803634790c142816 2023-04-20 12:55:28,991 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/59b48a0303a74a00803634790c142816, entries=150, sequenceid=320, filesize=25.4 K 2023-04-20 12:55:28,992 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/B/d294268ab3c243bd919ce29a5ecbeac3 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/d294268ab3c243bd919ce29a5ecbeac3 2023-04-20 12:55:28,994 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/d294268ab3c243bd919ce29a5ecbeac3, entries=150, sequenceid=320, filesize=11.9 K 2023-04-20 12:55:28,995 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/C/4a9436b4ec074a6a95d397965003e8cb as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/4a9436b4ec074a6a95d397965003e8cb 2023-04-20 12:55:28,996 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:28,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:28,997 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/4a9436b4ec074a6a95d397965003e8cb, entries=150, sequenceid=320, filesize=11.9 K 2023-04-20 12:55:28,998 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for aeecfead619437b2701aee4eba741c09 in 1658ms, sequenceid=320, compaction requested=true 2023-04-20 12:55:28,998 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for aeecfead619437b2701aee4eba741c09: 2023-04-20 12:55:28,998 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:55:28,998 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:55:28,998 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:55:28,998 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:55:28,998 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:55:28,999 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78684 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:55:28,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:28,999 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): aeecfead619437b2701aee4eba741c09/A is initiating minor compaction (all files) 2023-04-20 12:55:28,999 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of aeecfead619437b2701aee4eba741c09/A in TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:28,999 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:55:28,999 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): aeecfead619437b2701aee4eba741c09/B is initiating minor compaction (all files) 2023-04-20 12:55:28,999 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of aeecfead619437b2701aee4eba741c09/B in TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:28,999 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/b246ac2492584758afef8d558c827473, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/70920a1acff04c1abc16de623c78650c, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/d294268ab3c243bd919ce29a5ecbeac3] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp, totalSize=36.2 K 2023-04-20 12:55:29,000 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting b246ac2492584758afef8d558c827473, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1681995322769 2023-04-20 12:55:28,999 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/7918f59bb9b448968c29ded269a99b68, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/eaa812e10aed4bc0b2d840b12e526078, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/59b48a0303a74a00803634790c142816] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp, totalSize=76.8 K 2023-04-20 12:55:29,000 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 70920a1acff04c1abc16de623c78650c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=307, earliestPutTs=1681995325151 2023-04-20 12:55:29,000 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 7918f59bb9b448968c29ded269a99b68, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1681995322769 2023-04-20 12:55:29,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:29,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:29,000 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:29,000 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:29,000 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting d294268ab3c243bd919ce29a5ecbeac3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1681995327337 2023-04-20 12:55:29,000 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:29,000 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting eaa812e10aed4bc0b2d840b12e526078, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=307, earliestPutTs=1681995325151 2023-04-20 12:55:29,000 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:29,001 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 59b48a0303a74a00803634790c142816, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1681995327337 2023-04-20 12:55:29,001 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:29,001 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:29,002 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:29,002 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:29,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:29,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:29,002 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:29,002 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:29,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:29,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:29,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:29,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:29,003 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:29,003 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:29,003 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:29,003 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:29,003 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:29,003 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:29,003 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:29,003 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:29,003 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:29,003 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:29,003 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing aeecfead619437b2701aee4eba741c09 3/3 column families, dataSize=140.89 KB heapSize=386.16 KB 2023-04-20 12:55:29,004 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=A 2023-04-20 12:55:29,004 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:55:29,004 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=B 2023-04-20 12:55:29,004 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:55:29,004 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=C 2023-04-20 12:55:29,004 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:55:29,009 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): aeecfead619437b2701aee4eba741c09#B#compaction#358 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:55:29,022 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): aeecfead619437b2701aee4eba741c09#A#compaction#359 average throughput is 1.41 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:55:29,026 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/B/3e969e65804545bb963dbcf388f91fa5 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/3e969e65804545bb963dbcf388f91fa5 2023-04-20 12:55:29,029 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in aeecfead619437b2701aee4eba741c09/B of aeecfead619437b2701aee4eba741c09 into 3e969e65804545bb963dbcf388f91fa5(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:55:29,029 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for aeecfead619437b2701aee4eba741c09: 2023-04-20 12:55:29,029 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09., storeName=aeecfead619437b2701aee4eba741c09/B, priority=13, startTime=1681995328998; duration=0sec 2023-04-20 12:55:29,029 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:55:29,029 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:55:29,030 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:55:29,030 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): aeecfead619437b2701aee4eba741c09/C is initiating minor compaction (all files) 2023-04-20 12:55:29,030 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of aeecfead619437b2701aee4eba741c09/C in TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:29,030 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/d47d16e61b9444a09af0f6f7631f24c2, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/405807dd8f264aad8298596224a8ff33, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/4a9436b4ec074a6a95d397965003e8cb] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp, totalSize=36.2 K 2023-04-20 12:55:29,030 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting d47d16e61b9444a09af0f6f7631f24c2, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1681995322769 2023-04-20 12:55:29,031 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 405807dd8f264aad8298596224a8ff33, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=307, earliestPutTs=1681995325151 2023-04-20 12:55:29,031 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 4a9436b4ec074a6a95d397965003e8cb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1681995327337 2023-04-20 12:55:29,034 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042093366710b3d9432693263b8c600c692c to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042093366710b3d9432693263b8c600c692c 2023-04-20 12:55:29,047 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): aeecfead619437b2701aee4eba741c09#C#compaction#361 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:55:29,459 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=354, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/A/eb76655c5fbc42bb8cfe2c4d686c1c9b 2023-04-20 12:55:29,466 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/C/5837398c65b54d37a11a166ae2d2e98b as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/5837398c65b54d37a11a166ae2d2e98b 2023-04-20 12:55:29,473 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=354 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/B/a724e4a68bf54a12aab2b42e2dd96ef1 2023-04-20 12:55:29,476 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in aeecfead619437b2701aee4eba741c09/C of aeecfead619437b2701aee4eba741c09 into 5837398c65b54d37a11a166ae2d2e98b(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:55:29,476 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for aeecfead619437b2701aee4eba741c09: 2023-04-20 12:55:29,476 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09., storeName=aeecfead619437b2701aee4eba741c09/C, priority=13, startTime=1681995328998; duration=0sec 2023-04-20 12:55:29,476 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:55:29,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on aeecfead619437b2701aee4eba741c09 2023-04-20 12:55:29,528 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:29,537 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:29,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35892 deadline: 1681995389535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:29,541 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:29,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35900 deadline: 1681995389535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:29,542 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:29,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35842 deadline: 1681995389536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:29,542 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:29,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35848 deadline: 1681995389537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:29,542 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:29,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35886 deadline: 1681995389537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:29,642 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:29,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35892 deadline: 1681995389638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:29,643 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:29,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35900 deadline: 1681995389642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:29,644 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:29,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35842 deadline: 1681995389643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:29,644 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:29,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35848 deadline: 1681995389643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:29,644 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:29,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35886 deadline: 1681995389643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:29,847 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:29,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35892 deadline: 1681995389843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:29,848 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:29,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35848 deadline: 1681995389845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:29,848 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:29,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35900 deadline: 1681995389845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:29,849 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:29,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35842 deadline: 1681995389845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:29,850 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:29,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35886 deadline: 1681995389846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:29,860 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/A/bad10d61d37e4a02a5301574a086080f as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/bad10d61d37e4a02a5301574a086080f 2023-04-20 12:55:29,863 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in aeecfead619437b2701aee4eba741c09/A of aeecfead619437b2701aee4eba741c09 into bad10d61d37e4a02a5301574a086080f(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:55:29,863 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for aeecfead619437b2701aee4eba741c09: 2023-04-20 12:55:29,863 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09., storeName=aeecfead619437b2701aee4eba741c09/A, priority=13, startTime=1681995328998; duration=0sec 2023-04-20 12:55:29,863 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:55:29,888 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=354 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/C/712d4a53f23b40f089b3a165e4b64daf 2023-04-20 12:55:29,891 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/A/eb76655c5fbc42bb8cfe2c4d686c1c9b as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/eb76655c5fbc42bb8cfe2c4d686c1c9b 2023-04-20 12:55:29,894 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/eb76655c5fbc42bb8cfe2c4d686c1c9b, entries=150, sequenceid=354, filesize=25.4 K 2023-04-20 12:55:29,895 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/B/a724e4a68bf54a12aab2b42e2dd96ef1 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/a724e4a68bf54a12aab2b42e2dd96ef1 2023-04-20 12:55:29,898 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/a724e4a68bf54a12aab2b42e2dd96ef1, entries=150, sequenceid=354, filesize=11.9 K 2023-04-20 12:55:29,899 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/C/712d4a53f23b40f089b3a165e4b64daf as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/712d4a53f23b40f089b3a165e4b64daf 2023-04-20 12:55:29,901 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/712d4a53f23b40f089b3a165e4b64daf, entries=150, sequenceid=354, filesize=11.9 K 2023-04-20 12:55:29,902 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~386.11 KB/395376, currentSize=53.67 KB/54960 for aeecfead619437b2701aee4eba741c09 in 899ms, sequenceid=354, compaction requested=false 2023-04-20 12:55:29,902 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for aeecfead619437b2701aee4eba741c09: 2023-04-20 12:55:29,902 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:29,902 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:29,902 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:29,902 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:29,902 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:29,902 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:29,903 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:29,903 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:29,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:29,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:29,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:29,904 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:29,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:29,904 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:29,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:29,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:29,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:29,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:29,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:29,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e7d4059[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:29,905 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:29,905 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:29,905 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:29,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:29,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:29,905 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:29,905 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:29,905 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:29,905 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:29,905 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:29,906 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:29,906 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:29,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:29,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:29,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:29,906 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:29,906 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:29,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:29,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:29,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:29,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:29,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:29,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:29,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:29,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:29,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:29,908 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:29,908 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:29,908 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:29,908 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:29,908 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:29,909 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:29,909 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:29,909 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:29,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:29,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:29,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:29,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:29,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:29,909 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:29,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:29,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:29,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:29,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:29,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:29,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:29,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:29,910 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:29,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:29,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:29,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:29,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:29,912 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:29,912 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:29,912 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:29,912 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:29,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:29,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:29,912 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:29,912 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:29,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:29,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:29,912 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:29,913 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:29,913 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:29,913 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:29,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:29,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:29,913 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:29,913 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:29,913 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:30,013 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:30,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:30,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:30,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:30,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:30,016 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:30,016 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:30,016 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:30,016 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:30,016 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:30,016 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:30,017 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:30,017 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:30,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:30,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:30,017 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:30,017 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:30,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:30,018 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:30,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:30,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:30,018 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:30,018 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:30,018 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:30,018 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:30,018 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:30,018 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:30,022 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:30,022 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:30,022 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:30,022 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:30,023 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing aeecfead619437b2701aee4eba741c09 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-20 12:55:30,023 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=A 2023-04-20 12:55:30,023 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:30,023 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=B 2023-04-20 12:55:30,023 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:30,023 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=C 2023-04-20 12:55:30,023 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:30,034 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420698bc2d678cc4c4e81f6377ebcb935aa to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420698bc2d678cc4c4e81f6377ebcb935aa 2023-04-20 12:55:30,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on aeecfead619437b2701aee4eba741c09 2023-04-20 12:55:30,152 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:30,173 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:30,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35848 deadline: 1681995390168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:30,173 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681995330167, max timestamp=1681995330172Num uniques 859; 2023-04-20 12:55:30,173 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 12:55:30,173 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681995330166, max timestamp=1681995330172Num uniques 859; 2023-04-20 12:55:30,174 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681995330167, max timestamp=1681995330172Num uniques 859; 2023-04-20 12:55:30,174 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-20 12:55:30,174 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 12:55:30,224 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:30,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35900 deadline: 1681995390218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:30,224 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:30,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35892 deadline: 1681995390218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:30,228 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:30,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35842 deadline: 1681995390219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:30,230 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:30,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35886 deadline: 1681995390220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:30,277 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:30,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35848 deadline: 1681995390274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:30,328 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:30,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35900 deadline: 1681995390325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:30,328 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:30,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35892 deadline: 1681995390325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:30,332 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:30,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35842 deadline: 1681995390328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:30,334 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:30,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35886 deadline: 1681995390330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:30,438 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=368, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/A/1a178324df614132a3093e5da89238aa 2023-04-20 12:55:30,480 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:30,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35848 deadline: 1681995390479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:30,531 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:30,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35900 deadline: 1681995390530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:30,531 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:30,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35892 deadline: 1681995390530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:30,536 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:30,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35842 deadline: 1681995390533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:30,537 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:30,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35886 deadline: 1681995390536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:30,786 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:30,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35848 deadline: 1681995390782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:30,836 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:30,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35900 deadline: 1681995390832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:30,836 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:30,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35892 deadline: 1681995390832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:30,842 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:30,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35842 deadline: 1681995390837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:30,843 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:30,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35886 deadline: 1681995390839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:30,846 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=368 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/B/5a0b25d39a2440baaa803efde44d4923 2023-04-20 12:55:31,254 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=368 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/C/3e6fb0ab59b345359f2eb84b08d5834a 2023-04-20 12:55:31,258 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/A/1a178324df614132a3093e5da89238aa as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/1a178324df614132a3093e5da89238aa 2023-04-20 12:55:31,261 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/1a178324df614132a3093e5da89238aa, entries=150, sequenceid=368, filesize=25.4 K 2023-04-20 12:55:31,261 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/B/5a0b25d39a2440baaa803efde44d4923 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/5a0b25d39a2440baaa803efde44d4923 2023-04-20 12:55:31,264 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/5a0b25d39a2440baaa803efde44d4923, entries=150, sequenceid=368, filesize=11.9 K 2023-04-20 12:55:31,265 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/C/3e6fb0ab59b345359f2eb84b08d5834a as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/3e6fb0ab59b345359f2eb84b08d5834a 2023-04-20 12:55:31,268 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/3e6fb0ab59b345359f2eb84b08d5834a, entries=150, sequenceid=368, filesize=11.9 K 2023-04-20 12:55:31,269 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=125.23 KB/128240 for aeecfead619437b2701aee4eba741c09 in 1246ms, sequenceid=368, compaction requested=true 2023-04-20 12:55:31,269 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for aeecfead619437b2701aee4eba741c09: 2023-04-20 12:55:31,269 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:31,269 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:31,269 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:31,269 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,269 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:31,269 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:31,270 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,270 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:31,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:31,270 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,270 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:31,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:31,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:31,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:31,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:31,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:31,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7320929f[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:31,272 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:31,272 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,273 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,273 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,273 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:31,273 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:31,273 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:31,273 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:31,274 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,274 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:31,274 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:31,274 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:31,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:31,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:31,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:31,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:31,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:31,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:31,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:31,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:31,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:31,277 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:31,277 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:31,277 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:31,277 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:31,277 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:31,278 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,278 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:31,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:31,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,278 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:31,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:31,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:31,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:31,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:31,279 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:31,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:31,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:31,280 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:31,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,281 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,281 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:31,281 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:31,281 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:31,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:31,281 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:31,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:31,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:31,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:31,281 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:31,282 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,282 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,282 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:31,282 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,282 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:31,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:31,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:31,282 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on aeecfead619437b2701aee4eba741c09 2023-04-20 12:55:31,294 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing aeecfead619437b2701aee4eba741c09 3/3 column families, dataSize=131.94 KB heapSize=375.22 KB 2023-04-20 12:55:31,294 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=A 2023-04-20 12:55:31,294 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:55:31,294 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=B 2023-04-20 12:55:31,294 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:55:31,294 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=C 2023-04-20 12:55:31,294 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:55:31,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:31,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35848 deadline: 1681995391333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,349 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:31,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35900 deadline: 1681995391341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,349 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:31,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35892 deadline: 1681995391342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,350 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:31,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35842 deadline: 1681995391343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,350 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:31,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35886 deadline: 1681995391346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,382 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:31,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:31,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:31,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:31,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:31,385 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:31,385 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:31,385 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:31,385 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:31,385 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,385 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:31,386 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:31,386 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:31,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:31,386 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,386 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:31,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:31,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:31,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:31,387 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:31,387 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:31,387 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:31,387 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:31,387 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:31,387 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:31,387 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:31,387 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:31,387 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:31,387 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:31,387 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:31,387 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:31,388 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:31,388 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:31,388 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,388 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:31,388 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:31,388 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,388 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:31,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:31,389 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,389 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:31,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:31,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:31,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:31,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:31,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:31,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6853ed8[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:31,390 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:31,390 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,390 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,390 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:31,390 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,390 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:31,390 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:31,390 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:31,391 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:31,391 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:31,391 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:31,391 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:31,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:31,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:31,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:31,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:31,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:31,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:31,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:31,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:31,393 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:31,393 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:31,393 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:31,393 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:31,393 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:31,394 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,394 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:31,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,402 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:31,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:31,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:31,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:31,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:31,402 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:31,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:31,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:31,403 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:31,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:31,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,407 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:31,407 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,407 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:31,407 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:31,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:31,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:31,407 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:31,407 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:31,407 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:31,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:31,407 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:31,407 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,407 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:31,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:31,407 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,408 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,408 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:31,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:31,455 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:31,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35848 deadline: 1681995391440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,508 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:31,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:31,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:31,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:31,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:31,510 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:31,510 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:31,510 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:31,510 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:31,511 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,511 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:31,512 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,512 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:31,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:31,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:31,512 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,512 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:31,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:31,512 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:31,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:31,512 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:31,512 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:31,513 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:31,513 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:31,513 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:31,514 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:31,514 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:31,514 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:31,514 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:31,514 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:31,514 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:31,514 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:31,514 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:31,514 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,514 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:31,514 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:31,515 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,515 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:31,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:31,515 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,515 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:31,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:31,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:31,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:31,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:31,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:31,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@513bd65d[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:31,516 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:31,516 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,517 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,517 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:31,517 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:31,517 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,517 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:31,517 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:31,518 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,518 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:31,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:31,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:31,518 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:31,518 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:31,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:31,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:31,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:31,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:31,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:31,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:31,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:31,520 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:31,520 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:31,520 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:31,520 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:31,520 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:31,520 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:31,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:31,520 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,521 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:31,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:31,521 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:31,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:31,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:31,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:31,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:31,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,522 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:31,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:31,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,523 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,523 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:31,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:31,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:31,523 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:31,523 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:31,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:31,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:31,523 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:31,523 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:31,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:31,523 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,524 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,524 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:31,524 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,524 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:31,524 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:31,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:31,624 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:31,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:31,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:31,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:31,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:31,626 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:31,626 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:31,626 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:31,626 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:31,627 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,627 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:31,628 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,628 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:31,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:31,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:31,628 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,628 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:31,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:31,629 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:31,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:31,629 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:31,629 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:31,629 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:31,629 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:31,629 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:31,629 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:31,629 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:31,629 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:31,629 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:31,629 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:31,629 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:31,630 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:31,630 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:31,630 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:31,630 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,630 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:31,630 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:31,630 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,630 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:31,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:31,631 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,631 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:31,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:31,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:31,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:31,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:31,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:31,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55c03bab[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:31,632 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:31,632 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,632 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,632 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:31,632 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:31,632 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:31,632 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,632 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:31,633 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:31,633 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,633 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:31,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:31,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:31,633 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:31,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:31,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:31,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:31,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:31,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:31,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:31,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:31,634 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:31,634 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:31,635 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:31,635 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:31,635 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:31,636 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,636 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:31,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:31,636 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:31,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,637 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:31,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:31,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:31,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:31,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:31,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:31,637 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:31,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:31,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,639 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,639 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:31,639 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:31,639 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:31,639 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:31,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:31,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:31,639 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:31,639 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:31,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:31,639 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:31,639 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:31,639 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,639 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:31,639 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,639 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:31,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:31,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:31,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35848 deadline: 1681995391656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,706 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420490c4432c7aa48e692564e8c5e29855f to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420490c4432c7aa48e692564e8c5e29855f 2023-04-20 12:55:31,739 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:31,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:31,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:31,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:31,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:31,742 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:31,742 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:31,742 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:31,742 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:31,742 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,742 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:31,743 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,743 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:31,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:31,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:31,743 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,743 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:31,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:31,744 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:31,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:31,744 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:31,744 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:31,744 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:31,744 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:31,744 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:31,744 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:31,744 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:31,744 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:31,744 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:31,744 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:31,745 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:31,745 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:31,745 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:31,745 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:31,745 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,745 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:31,745 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:31,745 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,745 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:31,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:31,746 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,746 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:31,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:31,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:31,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:31,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:31,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:31,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2203efe6[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:31,747 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:31,747 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,748 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,749 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:31,749 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:31,749 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:31,749 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,749 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:31,749 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,749 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:31,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:31,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:31,749 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:31,749 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:31,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:31,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:31,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:31,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:31,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:31,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:31,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:31,751 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:31,751 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:31,751 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:31,751 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:31,751 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:31,753 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,753 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:31,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:31,753 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:31,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:31,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:31,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:31,754 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:31,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:31,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:31,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,756 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:31,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:31,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,759 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:31,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:31,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:31,759 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:31,759 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:31,760 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,760 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:31,760 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:31,760 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,760 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,760 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,760 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,760 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:31,760 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:31,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:31,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:31,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:31,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:31,860 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:31,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:31,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:31,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:31,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:31,862 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:31,862 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:31,862 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:31,862 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:31,863 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,863 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:31,863 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:31,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:31,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:31,863 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:31,863 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,864 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:31,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:31,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:31,864 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:31,864 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:31,864 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:31,864 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:31,864 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:31,864 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:31,866 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:31,866 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:31,866 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:31,866 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:31,866 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:31,866 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:31,866 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:31,866 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:31,866 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,866 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:31,866 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:31,867 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,867 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:31,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:31,867 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,867 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:31,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:31,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:31,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:31,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:31,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:31,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ddf6680[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:31,868 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:31,868 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,869 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,869 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:31,869 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:31,869 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,869 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:31,869 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:31,870 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:31,870 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,870 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:31,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:31,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:31,870 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:31,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:31,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:31,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:31,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:31,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:31,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:31,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:31,873 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:31,873 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:31,873 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:31,873 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:31,873 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:31,874 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,874 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:31,874 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:31,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:31,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:31,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:31,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:31,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,875 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:31,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:31,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:31,876 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:31,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:31,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,879 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,879 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:31,879 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:31,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:31,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:31,879 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:31,879 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:31,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:31,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:31,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:31,879 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:31,879 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,879 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,879 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:31,879 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:31,879 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,879 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:31,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:31,961 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:31,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35848 deadline: 1681995391960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,979 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:31,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:31,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:31,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:31,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:31,982 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:31,982 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:31,982 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:31,982 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:31,982 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,982 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:31,983 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,983 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:31,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:31,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:31,983 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,983 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:31,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:31,984 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:31,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:31,984 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:31,984 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:31,984 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:31,984 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:31,984 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:31,984 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:31,984 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:31,984 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:31,984 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:31,984 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:31,984 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:31,984 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:31,985 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:31,985 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:31,985 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,985 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:31,985 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:31,985 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,985 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:31,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:31,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:31,986 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,986 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:31,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:31,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:31,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:31,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:31,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@259c058[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:31,987 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:31,987 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,988 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,988 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:31,988 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:31,988 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:31,990 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,990 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:31,990 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:31,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:31,990 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:31,990 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:31,990 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:31,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:31,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:31,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:31,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:31,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:31,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:31,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:31,992 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:31,992 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:31,992 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:31,992 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:31,992 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:31,993 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,993 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,993 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:31,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:31,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:31,993 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:31,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:31,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:31,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:31,994 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:31,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:31,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:31,995 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,995 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:31,996 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:31,996 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:31,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:31,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:31,996 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:31,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:31,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:31,996 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:31,996 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:31,996 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:31,996 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:31,996 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:31,996 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,996 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:31,996 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:31,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:31,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:32,096 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:32,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:32,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:32,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:32,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:32,099 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:32,099 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:32,099 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:32,099 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:32,099 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:32,099 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:32,100 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:32,100 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:32,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:32,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:32,100 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:32,100 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:32,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:32,100 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:32,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:32,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:32,101 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:32,101 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:32,101 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:32,101 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:32,101 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:32,101 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:32,101 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:32,101 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:32,101 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:32,101 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:32,101 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:32,101 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:32,101 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:32,101 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:32,101 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:32,101 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:32,101 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:32,102 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:32,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:32,102 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:32,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:32,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:32,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:32,102 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:32,102 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:32,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:32,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:32,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:32,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:32,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:32,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5610aedb[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:32,103 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:32,103 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:32,104 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:32,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:32,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:32,104 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:32,104 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:32,104 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:32,104 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:32,104 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:32,105 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:32,105 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:32,105 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:32,105 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:32,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:32,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:32,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:32,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:32,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:32,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:32,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:32,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:32,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:32,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:32,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:32,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:32,107 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:32,107 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:32,107 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:32,107 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:32,107 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:32,108 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:32,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:32,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:32,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:32,108 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:32,108 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:32,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:32,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:32,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:32,108 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:32,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:32,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:32,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:32,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:32,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:32,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:32,109 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:32,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:32,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:32,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:32,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:32,111 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=404, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/A/afe7482ab4cd44038debea15340a588e 2023-04-20 12:55:32,111 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:32,111 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:32,111 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:32,111 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:32,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:32,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:32,111 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:32,111 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:32,111 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:32,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:32,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:32,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:32,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:32,111 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:32,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:32,111 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:32,112 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:32,113 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:32,113 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:32,213 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:32,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:32,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:32,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:32,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:32,215 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:32,215 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:32,215 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:32,216 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:32,216 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:32,216 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:32,217 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:32,217 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:32,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:32,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:32,217 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:32,217 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:32,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:32,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:32,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:32,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:32,218 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:32,218 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:32,218 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:32,218 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:32,218 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:32,218 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:32,218 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:32,218 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:32,218 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:32,218 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. started... 2023-04-20 12:55:32,218 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. as already flushing 2023-04-20 12:55:32,218 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:32,218 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:32,218 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:32,218 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:32,218 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:32,219 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:32,219 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:32,219 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:32,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:32,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:32,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:32,220 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:32,220 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:32,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:32,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:32,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:32,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:32,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:32,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:32,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b511720[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:32,221 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:32,221 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:32,221 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:32,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:32,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:32,221 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:32,222 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:32,222 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:32,222 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:32,222 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:32,222 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:32,222 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:32,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:32,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:32,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:32,222 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:32,222 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:32,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:32,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:32,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:32,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:32,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:32,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:32,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:32,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:32,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:32,224 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:32,224 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:32,224 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:32,224 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:32,224 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:32,225 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:32,225 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:32,225 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:32,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:32,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:32,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:32,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:32,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:32,225 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:32,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:32,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:32,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:32,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:32,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:32,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:32,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:32,226 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:32,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:32,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:32,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:32,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:32,227 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:32,227 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:32,227 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:32,227 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:32,227 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:32,228 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:32,228 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:32,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:32,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:32,228 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:32,228 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:32,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:32,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:32,228 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:32,228 DEBUG [Thread-2358] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:32,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:32,228 DEBUG [Thread-2358] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:32,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:32,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:32,248 DEBUG [Thread-2361] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x525c59d1 to 127.0.0.1:56673 2023-04-20 12:55:32,248 DEBUG [Thread-2361] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:55:32,253 DEBUG [Thread-2365] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1747f9fc to 127.0.0.1:56673 2023-04-20 12:55:32,253 DEBUG [Thread-2365] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:55:32,253 DEBUG [Thread-2367] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3098b23b to 127.0.0.1:56673 2023-04-20 12:55:32,253 DEBUG [Thread-2367] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:55:32,255 DEBUG [Thread-2363] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x078ba718 to 127.0.0.1:56673 2023-04-20 12:55:32,255 DEBUG [Thread-2363] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:55:32,260 DEBUG [Thread-2359] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x144047c5 to 127.0.0.1:56673 2023-04-20 12:55:32,260 DEBUG [Thread-2359] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:55:32,328 DEBUG [Thread-2358] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:32,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:32,355 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:32,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35900 deadline: 1681995392355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:32,355 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:32,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35886 deadline: 1681995392355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:32,358 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:32,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35842 deadline: 1681995392358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:32,360 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:32,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35892 deadline: 1681995392360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:32,466 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:32,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:35848 deadline: 1681995392465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:32,528 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=404 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/B/0e3efce276ae4a0aaf0ba14218973193 2023-04-20 12:55:32,945 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=404 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/C/64cec73b7e2b408d8b64096c13638373 2023-04-20 12:55:32,954 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/A/afe7482ab4cd44038debea15340a588e as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/afe7482ab4cd44038debea15340a588e 2023-04-20 12:55:32,960 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/afe7482ab4cd44038debea15340a588e, entries=200, sequenceid=404, filesize=32.3 K 2023-04-20 12:55:32,961 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/B/0e3efce276ae4a0aaf0ba14218973193 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/0e3efce276ae4a0aaf0ba14218973193 2023-04-20 12:55:32,965 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/0e3efce276ae4a0aaf0ba14218973193, entries=150, sequenceid=404, filesize=11.9 K 2023-04-20 12:55:32,966 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/C/64cec73b7e2b408d8b64096c13638373 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/64cec73b7e2b408d8b64096c13638373 2023-04-20 12:55:32,970 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/64cec73b7e2b408d8b64096c13638373, entries=150, sequenceid=404, filesize=11.9 K 2023-04-20 12:55:32,971 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~131.94 KB/135110, heapSize ~375.17 KB/384176, currentSize=46.96 KB/48090 for aeecfead619437b2701aee4eba741c09 in 1677ms, sequenceid=404, compaction requested=true 2023-04-20 12:55:32,971 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for aeecfead619437b2701aee4eba741c09: 2023-04-20 12:55:32,972 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:55:32,972 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 12:55:32,972 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 12:55:32,972 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:55:32,972 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:55:32,973 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 111872 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 12:55:32,973 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): aeecfead619437b2701aee4eba741c09/A is initiating minor compaction (all files) 2023-04-20 12:55:32,973 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49392 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 12:55:32,973 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of aeecfead619437b2701aee4eba741c09/A in TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:32,973 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): aeecfead619437b2701aee4eba741c09/B is initiating minor compaction (all files) 2023-04-20 12:55:32,973 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/bad10d61d37e4a02a5301574a086080f, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/eb76655c5fbc42bb8cfe2c4d686c1c9b, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/1a178324df614132a3093e5da89238aa, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/afe7482ab4cd44038debea15340a588e] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp, totalSize=109.3 K 2023-04-20 12:55:32,973 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of aeecfead619437b2701aee4eba741c09/B in TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:32,974 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/3e969e65804545bb963dbcf388f91fa5, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/a724e4a68bf54a12aab2b42e2dd96ef1, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/5a0b25d39a2440baaa803efde44d4923, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/0e3efce276ae4a0aaf0ba14218973193] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp, totalSize=48.2 K 2023-04-20 12:55:32,974 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting bad10d61d37e4a02a5301574a086080f, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1681995327337 2023-04-20 12:55:32,974 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 3e969e65804545bb963dbcf388f91fa5, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1681995327337 2023-04-20 12:55:32,974 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting eb76655c5fbc42bb8cfe2c4d686c1c9b, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=354, earliestPutTs=1681995327386 2023-04-20 12:55:32,974 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting a724e4a68bf54a12aab2b42e2dd96ef1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=354, earliestPutTs=1681995327386 2023-04-20 12:55:32,974 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 1a178324df614132a3093e5da89238aa, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1681995329534 2023-04-20 12:55:32,975 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 5a0b25d39a2440baaa803efde44d4923, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1681995329534 2023-04-20 12:55:32,975 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting afe7482ab4cd44038debea15340a588e, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1681995330216 2023-04-20 12:55:32,975 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 0e3efce276ae4a0aaf0ba14218973193, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1681995330216 2023-04-20 12:55:32,984 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): aeecfead619437b2701aee4eba741c09#B#compaction#371 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:55:32,986 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): aeecfead619437b2701aee4eba741c09#A#compaction#370 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:55:33,398 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/B/e94fb134ee8d41939e2b97619e2cdc66 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/e94fb134ee8d41939e2b97619e2cdc66 2023-04-20 12:55:33,404 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in aeecfead619437b2701aee4eba741c09/B of aeecfead619437b2701aee4eba741c09 into e94fb134ee8d41939e2b97619e2cdc66(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:55:33,404 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for aeecfead619437b2701aee4eba741c09: 2023-04-20 12:55:33,405 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09., storeName=aeecfead619437b2701aee4eba741c09/B, priority=12, startTime=1681995332972; duration=0sec 2023-04-20 12:55:33,405 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:55:33,405 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 12:55:33,407 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49392 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 12:55:33,407 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): aeecfead619437b2701aee4eba741c09/C is initiating minor compaction (all files) 2023-04-20 12:55:33,407 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of aeecfead619437b2701aee4eba741c09/C in TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:33,407 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/5837398c65b54d37a11a166ae2d2e98b, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/712d4a53f23b40f089b3a165e4b64daf, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/3e6fb0ab59b345359f2eb84b08d5834a, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/64cec73b7e2b408d8b64096c13638373] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp, totalSize=48.2 K 2023-04-20 12:55:33,407 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 5837398c65b54d37a11a166ae2d2e98b, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1681995327337 2023-04-20 12:55:33,408 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 712d4a53f23b40f089b3a165e4b64daf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=354, earliestPutTs=1681995327386 2023-04-20 12:55:33,408 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 3e6fb0ab59b345359f2eb84b08d5834a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1681995329534 2023-04-20 12:55:33,409 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 64cec73b7e2b408d8b64096c13638373, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1681995330216 2023-04-20 12:55:33,415 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): aeecfead619437b2701aee4eba741c09#C#compaction#372 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:55:33,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on aeecfead619437b2701aee4eba741c09 2023-04-20 12:55:33,472 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing aeecfead619437b2701aee4eba741c09 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-20 12:55:33,472 DEBUG [Thread-2350] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x48681e8f to 127.0.0.1:56673 2023-04-20 12:55:33,472 DEBUG [Thread-2350] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:55:33,472 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=A 2023-04-20 12:55:33,472 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:33,472 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=B 2023-04-20 12:55:33,472 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:33,472 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=C 2023-04-20 12:55:33,472 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:33,812 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/A/e54792aacc20406a87661554f27c9f0a as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/e54792aacc20406a87661554f27c9f0a 2023-04-20 12:55:33,816 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in aeecfead619437b2701aee4eba741c09/A of aeecfead619437b2701aee4eba741c09 into e54792aacc20406a87661554f27c9f0a(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:55:33,816 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for aeecfead619437b2701aee4eba741c09: 2023-04-20 12:55:33,816 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09., storeName=aeecfead619437b2701aee4eba741c09/A, priority=12, startTime=1681995332972; duration=0sec 2023-04-20 12:55:33,816 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:55:33,823 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/C/1e00fb748d5d4154a152c99f54aa4282 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/1e00fb748d5d4154a152c99f54aa4282 2023-04-20 12:55:33,828 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in aeecfead619437b2701aee4eba741c09/C of aeecfead619437b2701aee4eba741c09 into 1e00fb748d5d4154a152c99f54aa4282(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:55:33,828 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for aeecfead619437b2701aee4eba741c09: 2023-04-20 12:55:33,828 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09., storeName=aeecfead619437b2701aee4eba741c09/C, priority=12, startTime=1681995332972; duration=0sec 2023-04-20 12:55:33,828 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:55:33,888 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042052371480cba146df875eb8ef5edde4db to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042052371480cba146df875eb8ef5edde4db 2023-04-20 12:55:34,299 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=416, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/A/6f2054bb9386464cab3efdb2ca1a2bbd 2023-04-20 12:55:34,366 DEBUG [Thread-2352] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x342c272f to 127.0.0.1:56673 2023-04-20 12:55:34,366 DEBUG [Thread-2352] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:55:34,369 DEBUG [Thread-2356] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1aec6811 to 127.0.0.1:56673 2023-04-20 12:55:34,369 DEBUG [Thread-2356] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:55:34,374 DEBUG [Thread-2348] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6f29a773 to 127.0.0.1:56673 2023-04-20 12:55:34,374 DEBUG [Thread-2348] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:55:34,374 DEBUG [Thread-2354] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4e90d228 to 127.0.0.1:56673 2023-04-20 12:55:34,374 DEBUG [Thread-2354] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:55:34,374 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-20 12:55:34,375 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(394): wrote 73 2023-04-20 12:55:34,375 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(394): wrote 78 2023-04-20 12:55:34,375 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(394): wrote 69 2023-04-20 12:55:34,375 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(394): wrote 53 2023-04-20 12:55:34,375 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(394): wrote 64 2023-04-20 12:55:34,375 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-20 12:55:34,375 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-20 12:55:34,375 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(402): scanned 1855 2023-04-20 12:55:34,375 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(403): verified 5563 rows 2023-04-20 12:55:34,375 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(402): scanned 1857 2023-04-20 12:55:34,375 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(403): verified 5571 rows 2023-04-20 12:55:34,375 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(402): scanned 1878 2023-04-20 12:55:34,376 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(403): verified 5632 rows 2023-04-20 12:55:34,376 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(402): scanned 1872 2023-04-20 12:55:34,376 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(403): verified 5616 rows 2023-04-20 12:55:34,376 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(402): scanned 1868 2023-04-20 12:55:34,376 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(403): verified 5601 rows 2023-04-20 12:55:34,376 INFO [Listener at localhost.localdomain/42205] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-20 12:55:34,376 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6a409627 to 127.0.0.1:56673 2023-04-20 12:55:34,376 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:55:34,377 INFO [Listener at localhost.localdomain/42205] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-20 12:55:34,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-20 12:55:34,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-20 12:55:34,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-20 12:55:34,383 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681995334383"}]},"ts":"1681995334383"} 2023-04-20 12:55:34,384 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-20 12:55:34,391 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-20 12:55:34,392 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=aeecfead619437b2701aee4eba741c09, UNASSIGN}] 2023-04-20 12:55:34,393 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=aeecfead619437b2701aee4eba741c09, UNASSIGN 2023-04-20 12:55:34,393 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=aeecfead619437b2701aee4eba741c09, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:34,394 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681995334393"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681995334393"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681995334393"}]},"ts":"1681995334393"} 2023-04-20 12:55:34,394 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947}] 2023-04-20 12:55:34,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-20 12:55:34,546 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close aeecfead619437b2701aee4eba741c09 2023-04-20 12:55:34,546 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing aeecfead619437b2701aee4eba741c09, disabling compactions & flushes 2023-04-20 12:55:34,546 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:34,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-20 12:55:34,713 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=416 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/B/5a937388b82f416fbb9de4da4e725e38 2023-04-20 12:55:34,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-20 12:55:35,125 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=416 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/C/20295f82e6234e8ca0cc5dc2dae79d0a 2023-04-20 12:55:35,129 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/A/6f2054bb9386464cab3efdb2ca1a2bbd as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/6f2054bb9386464cab3efdb2ca1a2bbd 2023-04-20 12:55:35,132 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/6f2054bb9386464cab3efdb2ca1a2bbd, entries=150, sequenceid=416, filesize=25.4 K 2023-04-20 12:55:35,132 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/B/5a937388b82f416fbb9de4da4e725e38 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/5a937388b82f416fbb9de4da4e725e38 2023-04-20 12:55:35,135 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/5a937388b82f416fbb9de4da4e725e38, entries=150, sequenceid=416, filesize=11.9 K 2023-04-20 12:55:35,136 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/C/20295f82e6234e8ca0cc5dc2dae79d0a as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/20295f82e6234e8ca0cc5dc2dae79d0a 2023-04-20 12:55:35,139 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/20295f82e6234e8ca0cc5dc2dae79d0a, entries=150, sequenceid=416, filesize=11.9 K 2023-04-20 12:55:35,139 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=26.84 KB/27480 for aeecfead619437b2701aee4eba741c09 in 1667ms, sequenceid=416, compaction requested=false 2023-04-20 12:55:35,139 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for aeecfead619437b2701aee4eba741c09: 2023-04-20 12:55:35,139 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:35,139 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:35,139 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. after waiting 0 ms 2023-04-20 12:55:35,139 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:35,139 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing aeecfead619437b2701aee4eba741c09 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-20 12:55:35,140 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=A 2023-04-20 12:55:35,140 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:35,140 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=B 2023-04-20 12:55:35,140 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:35,140 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK aeecfead619437b2701aee4eba741c09, store=C 2023-04-20 12:55:35,140 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:35,156 DEBUG [regionserver/jenkins-hbase20:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-20 12:55:35,179 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/eec2a36dffba485eb8ea3ae39f3bc619, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/99a9cb3d3752450fbd4ac7b9289c9c79, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/42bdeca960674cac8294e9fcccc3a9a3, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/77a1caf6b7ef4e9a8b60dc22adb122d3, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/633e06da32f34a27bd5efbbdd68f5b39, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/b67e4c73e8be43e9a0013560dc1049ce, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/d65522859f5e42f4b1f48d443f42831c, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/25998a9af000484a878b4f7e424ff560, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/c1ce1022ac414922971a4f264ddf8317, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/0d9970eff125410d9f7259568da65a3f, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/d1bdd99960734e19a03f7f0b2a311120, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/6517274f9a5d4c37a48ac0c3c50042d4, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/0261f2f8bf934ef49e53cd0b795110ab, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/3a1b8587a3614360b7b4b7becf759b65, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/04630b82119948859e7dfe3a98cef1b7, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/7918f59bb9b448968c29ded269a99b68, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/1ff2247e942a49f58c43f4676e604e56, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/eaa812e10aed4bc0b2d840b12e526078, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/bad10d61d37e4a02a5301574a086080f, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/59b48a0303a74a00803634790c142816, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/eb76655c5fbc42bb8cfe2c4d686c1c9b, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/1a178324df614132a3093e5da89238aa, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/afe7482ab4cd44038debea15340a588e] to archive 2023-04-20 12:55:35,180 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-20 12:55:35,182 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/eec2a36dffba485eb8ea3ae39f3bc619 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/eec2a36dffba485eb8ea3ae39f3bc619 2023-04-20 12:55:35,183 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/99a9cb3d3752450fbd4ac7b9289c9c79 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/99a9cb3d3752450fbd4ac7b9289c9c79 2023-04-20 12:55:35,184 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/42bdeca960674cac8294e9fcccc3a9a3 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/42bdeca960674cac8294e9fcccc3a9a3 2023-04-20 12:55:35,184 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/77a1caf6b7ef4e9a8b60dc22adb122d3 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/77a1caf6b7ef4e9a8b60dc22adb122d3 2023-04-20 12:55:35,185 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/633e06da32f34a27bd5efbbdd68f5b39 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/633e06da32f34a27bd5efbbdd68f5b39 2023-04-20 12:55:35,186 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/b67e4c73e8be43e9a0013560dc1049ce to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/b67e4c73e8be43e9a0013560dc1049ce 2023-04-20 12:55:35,187 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/d65522859f5e42f4b1f48d443f42831c to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/d65522859f5e42f4b1f48d443f42831c 2023-04-20 12:55:35,187 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/25998a9af000484a878b4f7e424ff560 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/25998a9af000484a878b4f7e424ff560 2023-04-20 12:55:35,188 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/c1ce1022ac414922971a4f264ddf8317 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/c1ce1022ac414922971a4f264ddf8317 2023-04-20 12:55:35,189 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/0d9970eff125410d9f7259568da65a3f to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/0d9970eff125410d9f7259568da65a3f 2023-04-20 12:55:35,190 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/d1bdd99960734e19a03f7f0b2a311120 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/d1bdd99960734e19a03f7f0b2a311120 2023-04-20 12:55:35,191 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/6517274f9a5d4c37a48ac0c3c50042d4 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/6517274f9a5d4c37a48ac0c3c50042d4 2023-04-20 12:55:35,192 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/0261f2f8bf934ef49e53cd0b795110ab to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/0261f2f8bf934ef49e53cd0b795110ab 2023-04-20 12:55:35,193 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/3a1b8587a3614360b7b4b7becf759b65 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/3a1b8587a3614360b7b4b7becf759b65 2023-04-20 12:55:35,194 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/04630b82119948859e7dfe3a98cef1b7 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/04630b82119948859e7dfe3a98cef1b7 2023-04-20 12:55:35,195 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/7918f59bb9b448968c29ded269a99b68 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/7918f59bb9b448968c29ded269a99b68 2023-04-20 12:55:35,196 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/1ff2247e942a49f58c43f4676e604e56 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/1ff2247e942a49f58c43f4676e604e56 2023-04-20 12:55:35,197 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/eaa812e10aed4bc0b2d840b12e526078 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/eaa812e10aed4bc0b2d840b12e526078 2023-04-20 12:55:35,198 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/bad10d61d37e4a02a5301574a086080f to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/bad10d61d37e4a02a5301574a086080f 2023-04-20 12:55:35,199 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/59b48a0303a74a00803634790c142816 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/59b48a0303a74a00803634790c142816 2023-04-20 12:55:35,200 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/eb76655c5fbc42bb8cfe2c4d686c1c9b to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/eb76655c5fbc42bb8cfe2c4d686c1c9b 2023-04-20 12:55:35,201 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/1a178324df614132a3093e5da89238aa to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/1a178324df614132a3093e5da89238aa 2023-04-20 12:55:35,201 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/afe7482ab4cd44038debea15340a588e to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/afe7482ab4cd44038debea15340a588e 2023-04-20 12:55:35,207 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/4b6c335ee8134907b2f4484d0eb9f3a1, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/42d5f85f8ac948e6bd7eef8c7142faa8, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/efc5aabd716e4153bd07fc63309dc66e, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/373728976e774018a06afe9223645f63, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/c6313b421da44e3ca49f62f6f4e7e8b6, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/fa5432b2608947fd8551a336c40455ca, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/236a5c59b0ad45a69c391fff0c96276d, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/342db4cb15b34a4a98415ecc6331bf66, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/adcf302345f84faea891e9565e0f377a, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/70eb4f2936b647dea43220522ef33d65, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/618176180014400ab694fe6d0a86adf8, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/ec6d7cd665c749acb0b0b4b324a52f5a, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/5b7415cb7795431282c6834f9b0b741e, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/4420dd4bd54a4f7f834e4680f39dfa4a, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/4f3649c9ede34589a1dedcd58521730b, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/b246ac2492584758afef8d558c827473, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/e94fd2b13b5f4eb9a6245e28028a9222, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/70920a1acff04c1abc16de623c78650c, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/3e969e65804545bb963dbcf388f91fa5, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/d294268ab3c243bd919ce29a5ecbeac3, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/a724e4a68bf54a12aab2b42e2dd96ef1, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/5a0b25d39a2440baaa803efde44d4923, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/0e3efce276ae4a0aaf0ba14218973193] to archive 2023-04-20 12:55:35,207 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-20 12:55:35,208 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/4b6c335ee8134907b2f4484d0eb9f3a1 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/4b6c335ee8134907b2f4484d0eb9f3a1 2023-04-20 12:55:35,209 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/42d5f85f8ac948e6bd7eef8c7142faa8 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/42d5f85f8ac948e6bd7eef8c7142faa8 2023-04-20 12:55:35,210 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/efc5aabd716e4153bd07fc63309dc66e to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/efc5aabd716e4153bd07fc63309dc66e 2023-04-20 12:55:35,211 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/373728976e774018a06afe9223645f63 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/373728976e774018a06afe9223645f63 2023-04-20 12:55:35,212 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/c6313b421da44e3ca49f62f6f4e7e8b6 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/c6313b421da44e3ca49f62f6f4e7e8b6 2023-04-20 12:55:35,212 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/fa5432b2608947fd8551a336c40455ca to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/fa5432b2608947fd8551a336c40455ca 2023-04-20 12:55:35,213 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/236a5c59b0ad45a69c391fff0c96276d to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/236a5c59b0ad45a69c391fff0c96276d 2023-04-20 12:55:35,214 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/342db4cb15b34a4a98415ecc6331bf66 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/342db4cb15b34a4a98415ecc6331bf66 2023-04-20 12:55:35,214 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/adcf302345f84faea891e9565e0f377a to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/adcf302345f84faea891e9565e0f377a 2023-04-20 12:55:35,215 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/70eb4f2936b647dea43220522ef33d65 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/70eb4f2936b647dea43220522ef33d65 2023-04-20 12:55:35,216 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/618176180014400ab694fe6d0a86adf8 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/618176180014400ab694fe6d0a86adf8 2023-04-20 12:55:35,216 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/ec6d7cd665c749acb0b0b4b324a52f5a to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/ec6d7cd665c749acb0b0b4b324a52f5a 2023-04-20 12:55:35,217 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/5b7415cb7795431282c6834f9b0b741e to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/5b7415cb7795431282c6834f9b0b741e 2023-04-20 12:55:35,218 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/4420dd4bd54a4f7f834e4680f39dfa4a to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/4420dd4bd54a4f7f834e4680f39dfa4a 2023-04-20 12:55:35,219 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/4f3649c9ede34589a1dedcd58521730b to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/4f3649c9ede34589a1dedcd58521730b 2023-04-20 12:55:35,219 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/b246ac2492584758afef8d558c827473 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/b246ac2492584758afef8d558c827473 2023-04-20 12:55:35,220 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/e94fd2b13b5f4eb9a6245e28028a9222 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/e94fd2b13b5f4eb9a6245e28028a9222 2023-04-20 12:55:35,221 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/70920a1acff04c1abc16de623c78650c to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/70920a1acff04c1abc16de623c78650c 2023-04-20 12:55:35,222 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/3e969e65804545bb963dbcf388f91fa5 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/3e969e65804545bb963dbcf388f91fa5 2023-04-20 12:55:35,223 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/d294268ab3c243bd919ce29a5ecbeac3 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/d294268ab3c243bd919ce29a5ecbeac3 2023-04-20 12:55:35,224 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/a724e4a68bf54a12aab2b42e2dd96ef1 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/a724e4a68bf54a12aab2b42e2dd96ef1 2023-04-20 12:55:35,224 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/5a0b25d39a2440baaa803efde44d4923 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/5a0b25d39a2440baaa803efde44d4923 2023-04-20 12:55:35,225 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/0e3efce276ae4a0aaf0ba14218973193 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/0e3efce276ae4a0aaf0ba14218973193 2023-04-20 12:55:35,229 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/514cca0401b24ecb87cbb3c0c4a5f63a, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/4ca7b4e2c47c4fbdaefd77487f4884c0, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/4bab9dee8c3a45cdb6958e797f410910, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/a9dd16fe8dcb45aa8743a9c382fd94d9, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/d7c0f9df03e4416c8f865b607caa0f45, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/3294920550ff484a8b148c448289d901, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/a0721b80402b4bd59df816c3a8950731, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/6d6f869f0ad14e2b8d58f3d0cf7da2a6, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/5acdc459d92e40b291c6a4b0ab067365, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/c1f39d61f63b44068846f4978f1764c4, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/e8df2ce2b8d443249dac7fee27fb8157, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/5d3f4c05b20d4528a9a0d17f013af497, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/8169a02f389446f7a783465051126740, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/a8d86a2e24f747a8bd2d01f968016666, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/e0aa0f8b65d34b6da6f9dadaa3cca114, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/d47d16e61b9444a09af0f6f7631f24c2, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/d470330bb8ac4347873843d2457be056, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/405807dd8f264aad8298596224a8ff33, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/5837398c65b54d37a11a166ae2d2e98b, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/4a9436b4ec074a6a95d397965003e8cb, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/712d4a53f23b40f089b3a165e4b64daf, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/3e6fb0ab59b345359f2eb84b08d5834a, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/64cec73b7e2b408d8b64096c13638373] to archive 2023-04-20 12:55:35,229 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-20 12:55:35,230 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/514cca0401b24ecb87cbb3c0c4a5f63a to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/514cca0401b24ecb87cbb3c0c4a5f63a 2023-04-20 12:55:35,231 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/4ca7b4e2c47c4fbdaefd77487f4884c0 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/4ca7b4e2c47c4fbdaefd77487f4884c0 2023-04-20 12:55:35,232 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/4bab9dee8c3a45cdb6958e797f410910 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/4bab9dee8c3a45cdb6958e797f410910 2023-04-20 12:55:35,233 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/a9dd16fe8dcb45aa8743a9c382fd94d9 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/a9dd16fe8dcb45aa8743a9c382fd94d9 2023-04-20 12:55:35,233 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/d7c0f9df03e4416c8f865b607caa0f45 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/d7c0f9df03e4416c8f865b607caa0f45 2023-04-20 12:55:35,234 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/3294920550ff484a8b148c448289d901 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/3294920550ff484a8b148c448289d901 2023-04-20 12:55:35,235 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/a0721b80402b4bd59df816c3a8950731 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/a0721b80402b4bd59df816c3a8950731 2023-04-20 12:55:35,236 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/6d6f869f0ad14e2b8d58f3d0cf7da2a6 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/6d6f869f0ad14e2b8d58f3d0cf7da2a6 2023-04-20 12:55:35,236 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/5acdc459d92e40b291c6a4b0ab067365 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/5acdc459d92e40b291c6a4b0ab067365 2023-04-20 12:55:35,237 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/c1f39d61f63b44068846f4978f1764c4 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/c1f39d61f63b44068846f4978f1764c4 2023-04-20 12:55:35,238 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/e8df2ce2b8d443249dac7fee27fb8157 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/e8df2ce2b8d443249dac7fee27fb8157 2023-04-20 12:55:35,239 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/5d3f4c05b20d4528a9a0d17f013af497 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/5d3f4c05b20d4528a9a0d17f013af497 2023-04-20 12:55:35,240 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/8169a02f389446f7a783465051126740 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/8169a02f389446f7a783465051126740 2023-04-20 12:55:35,241 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/a8d86a2e24f747a8bd2d01f968016666 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/a8d86a2e24f747a8bd2d01f968016666 2023-04-20 12:55:35,242 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/e0aa0f8b65d34b6da6f9dadaa3cca114 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/e0aa0f8b65d34b6da6f9dadaa3cca114 2023-04-20 12:55:35,243 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/d47d16e61b9444a09af0f6f7631f24c2 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/d47d16e61b9444a09af0f6f7631f24c2 2023-04-20 12:55:35,244 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/d470330bb8ac4347873843d2457be056 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/d470330bb8ac4347873843d2457be056 2023-04-20 12:55:35,245 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/405807dd8f264aad8298596224a8ff33 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/405807dd8f264aad8298596224a8ff33 2023-04-20 12:55:35,246 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/5837398c65b54d37a11a166ae2d2e98b to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/5837398c65b54d37a11a166ae2d2e98b 2023-04-20 12:55:35,247 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/4a9436b4ec074a6a95d397965003e8cb to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/4a9436b4ec074a6a95d397965003e8cb 2023-04-20 12:55:35,248 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/712d4a53f23b40f089b3a165e4b64daf to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/712d4a53f23b40f089b3a165e4b64daf 2023-04-20 12:55:35,249 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/3e6fb0ab59b345359f2eb84b08d5834a to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/3e6fb0ab59b345359f2eb84b08d5834a 2023-04-20 12:55:35,249 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/64cec73b7e2b408d8b64096c13638373 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/64cec73b7e2b408d8b64096c13638373 2023-04-20 12:55:35,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-20 12:55:35,555 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420d5acb628b57b41b290b091d8dfa466a1 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420d5acb628b57b41b290b091d8dfa466a1 2023-04-20 12:55:35,961 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=425, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/A/0e155212df7149f5a0aaba59e45c58aa 2023-04-20 12:55:36,371 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=425 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/B/a18b4aea549c4662b6aa2d7db506dad4 2023-04-20 12:55:36,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-20 12:55:36,790 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=425 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/C/ac49f6edbe8947ac84bb9e99cdee2e55 2023-04-20 12:55:36,800 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/A/0e155212df7149f5a0aaba59e45c58aa as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/0e155212df7149f5a0aaba59e45c58aa 2023-04-20 12:55:36,804 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/0e155212df7149f5a0aaba59e45c58aa, entries=150, sequenceid=425, filesize=25.4 K 2023-04-20 12:55:36,805 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/B/a18b4aea549c4662b6aa2d7db506dad4 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/a18b4aea549c4662b6aa2d7db506dad4 2023-04-20 12:55:36,808 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/a18b4aea549c4662b6aa2d7db506dad4, entries=150, sequenceid=425, filesize=11.9 K 2023-04-20 12:55:36,809 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/.tmp/C/ac49f6edbe8947ac84bb9e99cdee2e55 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/ac49f6edbe8947ac84bb9e99cdee2e55 2023-04-20 12:55:36,812 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/ac49f6edbe8947ac84bb9e99cdee2e55, entries=150, sequenceid=425, filesize=11.9 K 2023-04-20 12:55:36,812 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for aeecfead619437b2701aee4eba741c09 in 1673ms, sequenceid=425, compaction requested=true 2023-04-20 12:55:36,820 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/recovered.edits/428.seqid, newMaxSeqId=428, maxSeqId=4 2023-04-20 12:55:36,821 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09. 2023-04-20 12:55:36,821 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for aeecfead619437b2701aee4eba741c09: 2023-04-20 12:55:36,823 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed aeecfead619437b2701aee4eba741c09 2023-04-20 12:55:36,823 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=aeecfead619437b2701aee4eba741c09, regionState=CLOSED 2023-04-20 12:55:36,824 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681995336823"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681995336823"}]},"ts":"1681995336823"} 2023-04-20 12:55:36,826 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-04-20 12:55:36,826 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure aeecfead619437b2701aee4eba741c09, server=jenkins-hbase20.apache.org,36277,1681995213947 in 2.4310 sec 2023-04-20 12:55:36,827 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-04-20 12:55:36,827 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=aeecfead619437b2701aee4eba741c09, UNASSIGN in 2.4340 sec 2023-04-20 12:55:36,827 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681995336827"}]},"ts":"1681995336827"} 2023-04-20 12:55:36,828 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-20 12:55:36,829 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-20 12:55:36,830 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.4510 sec 2023-04-20 12:55:38,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-20 12:55:38,491 INFO [Listener at localhost.localdomain/42205] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-04-20 12:55:38,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-20 12:55:38,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 12:55:38,497 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 12:55:38,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-20 12:55:38,498 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 12:55:38,503 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09 2023-04-20 12:55:38,507 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A, FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B, FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C, FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/recovered.edits] 2023-04-20 12:55:38,511 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/0e155212df7149f5a0aaba59e45c58aa to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/0e155212df7149f5a0aaba59e45c58aa 2023-04-20 12:55:38,513 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/6f2054bb9386464cab3efdb2ca1a2bbd to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/6f2054bb9386464cab3efdb2ca1a2bbd 2023-04-20 12:55:38,516 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/e54792aacc20406a87661554f27c9f0a to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/A/e54792aacc20406a87661554f27c9f0a 2023-04-20 12:55:38,520 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/5a937388b82f416fbb9de4da4e725e38 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/5a937388b82f416fbb9de4da4e725e38 2023-04-20 12:55:38,522 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/a18b4aea549c4662b6aa2d7db506dad4 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/a18b4aea549c4662b6aa2d7db506dad4 2023-04-20 12:55:38,524 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/e94fb134ee8d41939e2b97619e2cdc66 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/B/e94fb134ee8d41939e2b97619e2cdc66 2023-04-20 12:55:38,527 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/1e00fb748d5d4154a152c99f54aa4282 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/1e00fb748d5d4154a152c99f54aa4282 2023-04-20 12:55:38,528 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/20295f82e6234e8ca0cc5dc2dae79d0a to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/20295f82e6234e8ca0cc5dc2dae79d0a 2023-04-20 12:55:38,530 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/ac49f6edbe8947ac84bb9e99cdee2e55 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/C/ac49f6edbe8947ac84bb9e99cdee2e55 2023-04-20 12:55:38,533 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/recovered.edits/428.seqid to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09/recovered.edits/428.seqid 2023-04-20 12:55:38,534 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/aeecfead619437b2701aee4eba741c09 2023-04-20 12:55:38,534 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-20 12:55:38,534 DEBUG [PEWorker-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-20 12:55:38,535 DEBUG [PEWorker-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-20 12:55:38,537 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304201cb25233b3b047be9fb1450d3f7f974f to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304201cb25233b3b047be9fb1450d3f7f974f 2023-04-20 12:55:38,539 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304202166760463ee48c7970d872b62952f9a to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304202166760463ee48c7970d872b62952f9a 2023-04-20 12:55:38,540 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304202ecb8d5d8a194852b1573ae28705aeee to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304202ecb8d5d8a194852b1573ae28705aeee 2023-04-20 12:55:38,541 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042042592e5513594dad90ba618c383ddd8d to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042042592e5513594dad90ba618c383ddd8d 2023-04-20 12:55:38,542 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042044cd7ff2c2e2415495a0b407a7c4d1c4 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042044cd7ff2c2e2415495a0b407a7c4d1c4 2023-04-20 12:55:38,543 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420490c4432c7aa48e692564e8c5e29855f to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420490c4432c7aa48e692564e8c5e29855f 2023-04-20 12:55:38,544 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042050461b49fffc4f17abe51489b7f15404 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042050461b49fffc4f17abe51489b7f15404 2023-04-20 12:55:38,546 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042052371480cba146df875eb8ef5edde4db to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042052371480cba146df875eb8ef5edde4db 2023-04-20 12:55:38,547 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304205af21a6a19684d1694680bd9793cfa55 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304205af21a6a19684d1694680bd9793cfa55 2023-04-20 12:55:38,548 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304205c7e5ad6a03847b6900690881fe31e9d to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304205c7e5ad6a03847b6900690881fe31e9d 2023-04-20 12:55:38,549 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420698bc2d678cc4c4e81f6377ebcb935aa to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420698bc2d678cc4c4e81f6377ebcb935aa 2023-04-20 12:55:38,550 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304206ad8a607aeff4c50955c313e1f5dd5b2 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304206ad8a607aeff4c50955c313e1f5dd5b2 2023-04-20 12:55:38,551 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304208412f6e5c7f043f9b1c2f23fffae5dda to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304208412f6e5c7f043f9b1c2f23fffae5dda 2023-04-20 12:55:38,552 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304208ac144173e7f4b35b4125c93b6636a5d to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304208ac144173e7f4b35b4125c93b6636a5d 2023-04-20 12:55:38,554 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042093366710b3d9432693263b8c600c692c to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042093366710b3d9432693263b8c600c692c 2023-04-20 12:55:38,555 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304209ad01c18305f483a8fc751543efafea2 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304209ad01c18305f483a8fc751543efafea2 2023-04-20 12:55:38,556 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420bbef27b98ed443e48a0c525d79ceabc2 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420bbef27b98ed443e48a0c525d79ceabc2 2023-04-20 12:55:38,557 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420cc9fc10d69024da3bdc50672406562fc to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420cc9fc10d69024da3bdc50672406562fc 2023-04-20 12:55:38,559 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420d5acb628b57b41b290b091d8dfa466a1 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420d5acb628b57b41b290b091d8dfa466a1 2023-04-20 12:55:38,560 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420d70aa6429eba4c2db5494818dcdd63a0 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420d70aa6429eba4c2db5494818dcdd63a0 2023-04-20 12:55:38,561 DEBUG [PEWorker-5] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-20 12:55:38,563 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 12:55:38,564 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-20 12:55:38,566 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-20 12:55:38,567 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 12:55:38,567 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-20 12:55:38,568 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681995338568"}]},"ts":"9223372036854775807"} 2023-04-20 12:55:38,569 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-20 12:55:38,569 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => aeecfead619437b2701aee4eba741c09, NAME => 'TestAcidGuarantees,,1681995309132.aeecfead619437b2701aee4eba741c09.', STARTKEY => '', ENDKEY => ''}] 2023-04-20 12:55:38,569 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-20 12:55:38,569 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681995338569"}]},"ts":"9223372036854775807"} 2023-04-20 12:55:38,571 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-20 12:55:38,572 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 12:55:38,572 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 79 msec 2023-04-20 12:55:38,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-20 12:55:38,599 INFO [Listener at localhost.localdomain/42205] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-04-20 12:55:38,613 INFO [Listener at localhost.localdomain/42205] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=235 (was 224) - Thread LEAK? -, OpenFileDescriptor=537 (was 537), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=305 (was 314), ProcessCount=167 (was 170), AvailableMemoryMB=9010 (was 9164) 2023-04-20 12:55:38,625 INFO [Listener at localhost.localdomain/42205] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=235, OpenFileDescriptor=537, MaxFileDescriptor=60000, SystemLoadAverage=305, ProcessCount=167, AvailableMemoryMB=9010 2023-04-20 12:55:38,627 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-20 12:55:38,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-20 12:55:38,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-20 12:55:38,630 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-20 12:55:38,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-04-20 12:55:38,631 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-20 12:55:38,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-20 12:55:38,632 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672 2023-04-20 12:55:38,632 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672 empty. 2023-04-20 12:55:38,632 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672 2023-04-20 12:55:38,633 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-20 12:55:38,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-20 12:55:38,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-20 12:55:39,041 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-20 12:55:39,044 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => bed4cc4c061ff4314d49ea7add6e3672, NAME => 'TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp 2023-04-20 12:55:39,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-20 12:55:39,454 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-20 12:55:39,454 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing bed4cc4c061ff4314d49ea7add6e3672, disabling compactions & flushes 2023-04-20 12:55:39,454 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:39,454 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:39,454 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. after waiting 0 ms 2023-04-20 12:55:39,454 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:39,454 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:39,455 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:55:39,460 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-20 12:55:39,462 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681995339462"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681995339462"}]},"ts":"1681995339462"} 2023-04-20 12:55:39,463 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-20 12:55:39,464 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-20 12:55:39,464 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681995339464"}]},"ts":"1681995339464"} 2023-04-20 12:55:39,464 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-20 12:55:39,467 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=bed4cc4c061ff4314d49ea7add6e3672, ASSIGN}] 2023-04-20 12:55:39,468 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=bed4cc4c061ff4314d49ea7add6e3672, ASSIGN 2023-04-20 12:55:39,468 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=bed4cc4c061ff4314d49ea7add6e3672, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,36277,1681995213947; forceNewPlan=false, retain=false 2023-04-20 12:55:39,621 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=bed4cc4c061ff4314d49ea7add6e3672, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:39,621 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681995339620"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681995339620"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681995339620"}]},"ts":"1681995339620"} 2023-04-20 12:55:39,624 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947}] 2023-04-20 12:55:39,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-20 12:55:39,779 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:39,779 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => bed4cc4c061ff4314d49ea7add6e3672, NAME => 'TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.', STARTKEY => '', ENDKEY => ''} 2023-04-20 12:55:39,780 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees bed4cc4c061ff4314d49ea7add6e3672 2023-04-20 12:55:39,780 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-20 12:55:39,780 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for bed4cc4c061ff4314d49ea7add6e3672 2023-04-20 12:55:39,780 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for bed4cc4c061ff4314d49ea7add6e3672 2023-04-20 12:55:39,782 INFO [StoreOpener-bed4cc4c061ff4314d49ea7add6e3672-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region bed4cc4c061ff4314d49ea7add6e3672 2023-04-20 12:55:39,784 DEBUG [StoreOpener-bed4cc4c061ff4314d49ea7add6e3672-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A 2023-04-20 12:55:39,784 DEBUG [StoreOpener-bed4cc4c061ff4314d49ea7add6e3672-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A 2023-04-20 12:55:39,784 INFO [StoreOpener-bed4cc4c061ff4314d49ea7add6e3672-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 12:55:39,785 INFO [StoreOpener-bed4cc4c061ff4314d49ea7add6e3672-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bed4cc4c061ff4314d49ea7add6e3672 columnFamilyName A 2023-04-20 12:55:39,786 INFO [StoreOpener-bed4cc4c061ff4314d49ea7add6e3672-1] regionserver.HStore(310): Store=bed4cc4c061ff4314d49ea7add6e3672/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 12:55:39,786 INFO [StoreOpener-bed4cc4c061ff4314d49ea7add6e3672-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region bed4cc4c061ff4314d49ea7add6e3672 2023-04-20 12:55:39,788 DEBUG [StoreOpener-bed4cc4c061ff4314d49ea7add6e3672-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B 2023-04-20 12:55:39,788 DEBUG [StoreOpener-bed4cc4c061ff4314d49ea7add6e3672-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B 2023-04-20 12:55:39,789 INFO [StoreOpener-bed4cc4c061ff4314d49ea7add6e3672-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 12:55:39,789 INFO [StoreOpener-bed4cc4c061ff4314d49ea7add6e3672-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bed4cc4c061ff4314d49ea7add6e3672 columnFamilyName B 2023-04-20 12:55:39,790 INFO [StoreOpener-bed4cc4c061ff4314d49ea7add6e3672-1] regionserver.HStore(310): Store=bed4cc4c061ff4314d49ea7add6e3672/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 12:55:39,790 INFO [StoreOpener-bed4cc4c061ff4314d49ea7add6e3672-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region bed4cc4c061ff4314d49ea7add6e3672 2023-04-20 12:55:39,792 DEBUG [StoreOpener-bed4cc4c061ff4314d49ea7add6e3672-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C 2023-04-20 12:55:39,792 DEBUG [StoreOpener-bed4cc4c061ff4314d49ea7add6e3672-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C 2023-04-20 12:55:39,792 INFO [StoreOpener-bed4cc4c061ff4314d49ea7add6e3672-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 12:55:39,792 INFO [StoreOpener-bed4cc4c061ff4314d49ea7add6e3672-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bed4cc4c061ff4314d49ea7add6e3672 columnFamilyName C 2023-04-20 12:55:39,793 INFO [StoreOpener-bed4cc4c061ff4314d49ea7add6e3672-1] regionserver.HStore(310): Store=bed4cc4c061ff4314d49ea7add6e3672/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 12:55:39,793 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:39,794 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672 2023-04-20 12:55:39,794 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672 2023-04-20 12:55:39,797 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-20 12:55:39,798 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for bed4cc4c061ff4314d49ea7add6e3672 2023-04-20 12:55:39,799 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-20 12:55:39,800 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened bed4cc4c061ff4314d49ea7add6e3672; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67785982, jitterRate=0.010089844465255737}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-20 12:55:39,800 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:55:39,800 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672., pid=49, masterSystemTime=1681995339776 2023-04-20 12:55:39,801 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:39,801 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:39,802 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=bed4cc4c061ff4314d49ea7add6e3672, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:39,802 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681995339801"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1681995339801"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681995339801"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681995339801"}]},"ts":"1681995339801"} 2023-04-20 12:55:39,803 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-04-20 12:55:39,803 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 in 178 msec 2023-04-20 12:55:39,804 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-04-20 12:55:39,804 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=bed4cc4c061ff4314d49ea7add6e3672, ASSIGN in 336 msec 2023-04-20 12:55:39,805 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-20 12:55:39,805 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681995339805"}]},"ts":"1681995339805"} 2023-04-20 12:55:39,806 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-20 12:55:39,807 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-20 12:55:39,808 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1790 sec 2023-04-20 12:55:40,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-20 12:55:40,738 INFO [Listener at localhost.localdomain/42205] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-04-20 12:55:40,739 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x0817e2e5 to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:55:40,745 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f7a8e75, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:55:40,747 DEBUG [Listener at localhost.localdomain/42205] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:55:40,750 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:47038, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:55:40,751 DEBUG [Listener at localhost.localdomain/42205] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-20 12:55:40,753 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:51174, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-20 12:55:40,755 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x4048dcb4 to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:55:40,766 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@777f1e92, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:55:40,766 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x4e7a5e9a to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:55:40,771 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@48d1969d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:55:40,772 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x62447900 to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:55:40,777 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7ed6755c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:55:40,777 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x519db5f2 to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:55:40,781 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@10505ab8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:55:40,782 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x5c9c1feb to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:55:40,786 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@609213af, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:55:40,786 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x1ee0ad26 to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:55:40,790 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6e9fda3e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:55:40,790 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x6acb8caa to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:55:40,803 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@71d99548, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:55:40,804 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x70263db3 to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:55:40,812 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@46be51c1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:55:40,812 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x25d2db45 to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:55:40,816 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@64d8b1e9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:55:40,816 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x2c2ce5cd to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:55:40,821 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1f970bfe, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:55:40,825 DEBUG [hconnection-0x47cf1588-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:55:40,827 DEBUG [hconnection-0x10738358-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:55:40,827 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:47054, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:55:40,827 DEBUG [hconnection-0x3cf0cff8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:55:40,828 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:47068, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:55:40,830 DEBUG [hconnection-0x13fe8efb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:55:40,830 DEBUG [hconnection-0xa0357c8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:55:40,830 DEBUG [hconnection-0x40c0811e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:55:40,832 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:47070, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:55:40,832 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:47082, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:55:40,833 DEBUG [hconnection-0x5b3bd79b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:55:40,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:40,833 DEBUG [hconnection-0x7fe33168-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:55:40,845 DEBUG [hconnection-0x30adb5ad-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:55:40,845 DEBUG [hconnection-0x2bf810dc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:55:40,848 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:47100, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:55:40,849 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54242, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:55:40,851 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:47084, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:55:40,851 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54244, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:55:40,851 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:47102, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:55:40,852 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54262, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:55:40,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:40,860 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bed4cc4c061ff4314d49ea7add6e3672 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-20 12:55:40,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on bed4cc4c061ff4314d49ea7add6e3672 2023-04-20 12:55:40,863 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=A 2023-04-20 12:55:40,863 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:40,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:40,863 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=B 2023-04-20 12:55:40,863 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:40,864 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:40,863 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:40,864 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:40,864 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=C 2023-04-20 12:55:40,864 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:40,864 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:40,864 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:40,864 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:40,865 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:40,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:40,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:40,865 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:40,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:40,865 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:40,865 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:40,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:40,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:40,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:40,866 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:40,872 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:40,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681995340869, max timestamp=1681995340870Num uniques 859; 2023-04-20 12:55:40,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54244 deadline: 1681995400871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:40,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 12:55:40,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681995340869, max timestamp=1681995340870Num uniques 859; 2023-04-20 12:55:40,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 12:55:40,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681995340869, max timestamp=1681995340870Num uniques 859; 2023-04-20 12:55:40,873 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-20 12:55:40,879 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:40,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54262 deadline: 1681995400878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:40,880 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:40,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54242 deadline: 1681995400879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:40,880 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:40,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47054 deadline: 1681995400880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:40,880 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:40,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47068 deadline: 1681995400880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:40,882 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:40,882 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:40,882 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:40,882 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:40,882 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:40,888 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:40,888 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:40,888 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:40,888 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:40,889 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:40,889 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:40,889 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:40,889 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:40,889 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:40,889 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:40,889 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:40,890 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:40,890 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:40,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:40,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:40,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:40,890 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:40,890 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:40,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:40,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:40,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:40,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:40,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:40,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:40,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1dcc87ac[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:40,891 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:40,891 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:40,892 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:40,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:40,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:40,892 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:40,892 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:40,892 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:40,892 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:40,892 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:40,893 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:40,893 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:40,893 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:40,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:40,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:40,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:40,893 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:40,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:40,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:40,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:40,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:40,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:40,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:40,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:40,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:40,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:40,895 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:40,895 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:40,895 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:40,895 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:40,895 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:40,895 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/eccfdf12947f4871a6967f33fdbd6513 2023-04-20 12:55:40,896 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:40,896 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:40,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:40,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:40,896 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:40,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:40,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:40,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:40,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:40,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:40,896 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:40,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:40,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:40,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:40,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:40,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:40,898 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:40,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:40,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:40,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:40,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:40,899 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:40,899 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:40,899 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:40,899 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:40,899 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:40,899 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:40,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:40,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:40,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:40,900 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:40,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:40,899 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:40,900 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:40,900 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:40,900 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:40,900 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:40,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:40,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:40,900 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:40,976 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:40,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54244 deadline: 1681995400973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:40,982 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:40,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54262 deadline: 1681995400980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:40,984 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:40,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54242 deadline: 1681995400981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:40,985 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:40,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47054 deadline: 1681995400981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:40,985 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:40,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47068 deadline: 1681995400981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,000 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:41,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:41,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:41,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:41,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:41,006 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:41,006 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:41,006 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:41,006 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:41,007 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:41,007 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:41,007 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:41,007 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:41,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:41,008 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:41,008 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:41,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:41,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:41,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:41,008 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:41,014 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:41,014 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:41,014 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:41,014 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:41,014 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:41,014 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:41,014 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:41,014 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:41,014 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:41,014 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:41,014 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:41,014 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:41,014 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:41,014 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,014 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:41,014 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:41,023 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:41,023 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:41,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:41,023 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:41,023 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:41,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:41,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:41,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:41,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:41,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:41,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51dbcf4d[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:41,024 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:41,024 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:41,027 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:41,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:41,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:41,027 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:41,028 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:41,028 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,028 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:41,028 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:41,028 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:41,028 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,028 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:41,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:41,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:41,028 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:41,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:41,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:41,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:41,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:41,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:41,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:41,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:41,030 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:41,030 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:41,030 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:41,030 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:41,030 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:41,031 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:41,031 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:41,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:41,031 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:41,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:41,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:41,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:41,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:41,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:41,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:41,032 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:41,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:41,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:41,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:41,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:41,032 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:41,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:41,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,034 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,034 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:41,034 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:41,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:41,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:41,034 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:41,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:41,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:41,034 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:41,034 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:41,034 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:41,034 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:41,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:41,034 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:41,034 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:41,034 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:41,034 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:41,034 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:41,134 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:41,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:41,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:41,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:41,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:41,137 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:41,137 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:41,137 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:41,137 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:41,137 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:41,137 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:41,138 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,138 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:41,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:41,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:41,138 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,138 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:41,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:41,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:41,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:41,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:41,139 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:41,139 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:41,139 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:41,139 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:41,139 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:41,139 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:41,139 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:41,139 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:41,139 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:41,139 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:41,139 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:41,139 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:41,139 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:41,139 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:41,140 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,140 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:41,140 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:41,140 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,140 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:41,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:41,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:41,141 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:41,141 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:41,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:41,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:41,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:41,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:41,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:41,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2eec5db[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:41,142 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:41,142 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:41,144 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:41,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:41,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:41,144 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:41,144 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:41,144 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:41,144 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,144 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:41,145 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,145 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:41,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:41,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:41,145 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:41,145 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:41,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:41,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:41,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:41,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:41,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:41,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:41,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:41,147 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:41,147 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:41,147 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:41,147 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:41,147 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:41,148 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:41,148 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:41,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:41,148 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:41,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:41,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:41,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:41,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:41,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:41,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:41,148 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:41,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:41,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:41,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:41,149 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:41,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:41,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:41,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,150 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:41,151 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:41,151 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:41,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:41,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:41,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:41,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:41,151 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,151 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:41,151 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:41,151 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,151 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:41,151 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:41,151 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:41,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:41,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:41,151 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:41,151 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:41,151 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:41,179 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:41,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54244 deadline: 1681995401177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,186 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:41,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54262 deadline: 1681995401183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,188 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:41,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47054 deadline: 1681995401186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,189 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:41,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47068 deadline: 1681995401186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,189 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:41,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54242 deadline: 1681995401186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,251 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:41,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:41,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:41,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:41,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:41,254 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:41,254 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:41,254 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:41,254 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:41,254 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:41,254 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:41,255 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,255 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:41,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:41,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:41,255 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,255 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:41,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:41,255 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:41,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:41,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:41,255 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:41,256 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:41,256 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:41,256 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:41,256 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:41,256 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:41,256 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:41,256 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:41,256 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:41,256 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:41,256 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:41,256 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:41,256 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:41,256 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:41,256 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,256 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:41,256 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:41,257 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,257 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:41,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:41,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:41,257 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:41,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:41,257 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:41,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:41,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:41,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:41,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:41,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61652ab1[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:41,258 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:41,258 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:41,259 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:41,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:41,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:41,259 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:41,259 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,259 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:41,259 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:41,259 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:41,260 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,260 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:41,260 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:41,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:41,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:41,260 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:41,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:41,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:41,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:41,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:41,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:41,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:41,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:41,261 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:41,261 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:41,262 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:41,262 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:41,262 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:41,263 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:41,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:41,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:41,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:41,264 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:41,265 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:41,265 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:41,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:41,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:41,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:41,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:41,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:41,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:41,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:41,266 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:41,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:41,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:41,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,281 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,282 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:41,282 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:41,282 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:41,282 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:41,282 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:41,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:41,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:41,282 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,282 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:41,282 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:41,282 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:41,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:41,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:41,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:41,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:41,282 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:41,282 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:41,282 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:41,315 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/9d68a7fcb1864ef3bdbc52c9b82459ec 2023-04-20 12:55:41,325 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/24a31044ddda451fbfe6116973c2411d 2023-04-20 12:55:41,328 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/eccfdf12947f4871a6967f33fdbd6513 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/eccfdf12947f4871a6967f33fdbd6513 2023-04-20 12:55:41,331 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/eccfdf12947f4871a6967f33fdbd6513, entries=150, sequenceid=12, filesize=11.6 K 2023-04-20 12:55:41,331 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/9d68a7fcb1864ef3bdbc52c9b82459ec as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/9d68a7fcb1864ef3bdbc52c9b82459ec 2023-04-20 12:55:41,334 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/9d68a7fcb1864ef3bdbc52c9b82459ec, entries=150, sequenceid=12, filesize=11.6 K 2023-04-20 12:55:41,334 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/24a31044ddda451fbfe6116973c2411d as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/24a31044ddda451fbfe6116973c2411d 2023-04-20 12:55:41,337 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/24a31044ddda451fbfe6116973c2411d, entries=150, sequenceid=12, filesize=11.6 K 2023-04-20 12:55:41,337 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for bed4cc4c061ff4314d49ea7add6e3672 in 477ms, sequenceid=12, compaction requested=false 2023-04-20 12:55:41,338 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:55:41,383 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:41,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:41,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:41,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:41,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:41,385 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:41,385 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:41,385 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:41,385 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:41,386 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:41,386 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:41,386 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,386 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:41,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:41,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:41,387 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,387 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:41,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:41,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:41,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:41,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:41,387 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:41,387 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:41,387 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:41,387 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:41,387 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:41,387 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:41,388 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:41,388 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:41,388 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:41,388 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:41,388 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bed4cc4c061ff4314d49ea7add6e3672 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-20 12:55:41,388 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=A 2023-04-20 12:55:41,388 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:55:41,388 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=B 2023-04-20 12:55:41,388 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:55:41,388 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=C 2023-04-20 12:55:41,388 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:55:41,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on bed4cc4c061ff4314d49ea7add6e3672 2023-04-20 12:55:41,484 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:41,494 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:41,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47068 deadline: 1681995401491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,494 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:41,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54242 deadline: 1681995401492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,495 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:41,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54262 deadline: 1681995401493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,495 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:41,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54244 deadline: 1681995401493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,498 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:41,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47054 deadline: 1681995401494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,600 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:41,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54262 deadline: 1681995401596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,600 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:41,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54244 deadline: 1681995401596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,600 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:41,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47054 deadline: 1681995401599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,794 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/fa56b98987e74bc6957d962f0890b1ce 2023-04-20 12:55:41,806 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:41,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54244 deadline: 1681995401801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,806 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:41,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54262 deadline: 1681995401802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,806 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:41,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47054 deadline: 1681995401802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:41,998 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:41,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54242 deadline: 1681995401996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:42,000 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:42,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47068 deadline: 1681995401996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:42,110 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:42,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54244 deadline: 1681995402107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:42,110 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:42,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54262 deadline: 1681995402107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:42,110 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:42,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47054 deadline: 1681995402107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:42,204 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/7c0222ce61b343d9b7849e5c37eff46a 2023-04-20 12:55:42,356 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-20 12:55:42,612 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/25d46b1616954b22ab16e3eb9cb8dd4f 2023-04-20 12:55:42,616 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:42,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47054 deadline: 1681995402612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:42,617 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:42,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54244 deadline: 1681995402613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:42,617 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:42,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54262 deadline: 1681995402613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:42,618 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/fa56b98987e74bc6957d962f0890b1ce as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/fa56b98987e74bc6957d962f0890b1ce 2023-04-20 12:55:42,621 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/fa56b98987e74bc6957d962f0890b1ce, entries=150, sequenceid=49, filesize=11.6 K 2023-04-20 12:55:42,622 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/7c0222ce61b343d9b7849e5c37eff46a as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/7c0222ce61b343d9b7849e5c37eff46a 2023-04-20 12:55:42,625 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/7c0222ce61b343d9b7849e5c37eff46a, entries=150, sequenceid=49, filesize=11.6 K 2023-04-20 12:55:42,626 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/25d46b1616954b22ab16e3eb9cb8dd4f as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/25d46b1616954b22ab16e3eb9cb8dd4f 2023-04-20 12:55:42,630 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/25d46b1616954b22ab16e3eb9cb8dd4f, entries=150, sequenceid=49, filesize=11.6 K 2023-04-20 12:55:42,630 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for bed4cc4c061ff4314d49ea7add6e3672 in 1242ms, sequenceid=49, compaction requested=false 2023-04-20 12:55:42,631 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:55:42,631 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:42,631 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:42,631 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:42,631 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:42,631 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:42,631 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:42,632 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:42,632 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:42,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:42,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:42,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:42,632 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:42,632 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:42,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:42,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:42,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:42,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:42,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:42,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:42,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@186d20d6[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:42,633 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:42,633 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:42,633 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:42,634 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:42,634 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:42,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:42,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:42,634 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:42,634 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:42,634 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:42,636 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:42,636 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:42,636 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:42,636 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:42,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:42,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:42,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:42,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:42,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:42,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:42,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:42,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:42,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:42,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:42,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:42,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:42,638 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:42,638 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:42,638 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:42,638 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:42,638 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:42,638 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:42,638 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:42,638 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:42,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:42,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:42,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:42,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:42,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:42,640 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:42,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:42,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:42,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:42,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:42,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:42,641 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:42,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:42,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:42,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:42,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:42,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:42,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:42,643 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:42,643 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:42,643 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:42,643 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:42,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:42,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:42,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:42,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:42,643 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:42,643 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:42,643 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:42,643 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:42,643 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:42,643 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:42,643 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:42,643 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:42,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:42,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:42,643 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:42,743 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:42,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:42,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:42,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:42,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:42,746 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:42,746 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:42,746 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:42,746 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:42,746 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:42,746 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:42,747 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:42,747 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:42,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:42,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:42,747 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:42,747 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:42,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:42,748 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:42,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:42,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:42,748 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:42,748 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:42,748 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:42,748 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:42,748 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:42,748 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:42,750 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:42,750 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:42,750 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:42,750 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:42,750 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bed4cc4c061ff4314d49ea7add6e3672 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-20 12:55:42,750 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=A 2023-04-20 12:55:42,750 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:42,750 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=B 2023-04-20 12:55:42,750 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:42,750 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=C 2023-04-20 12:55:42,750 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:43,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on bed4cc4c061ff4314d49ea7add6e3672 2023-04-20 12:55:43,000 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:43,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681995343036, max timestamp=1681995343043Num uniques 859; 2023-04-20 12:55:43,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 12:55:43,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681995343036, max timestamp=1681995343043Num uniques 859; 2023-04-20 12:55:43,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 12:55:43,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681995343036, max timestamp=1681995343043Num uniques 859; 2023-04-20 12:55:43,045 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 12:55:43,083 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:43,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54242 deadline: 1681995403077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,088 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:43,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47068 deadline: 1681995403083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,168 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/a4c07c0e94314eb8b28dfc257636743d 2023-04-20 12:55:43,186 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:43,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54242 deadline: 1681995403184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,192 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:43,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47068 deadline: 1681995403189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,396 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:43,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54242 deadline: 1681995403387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,397 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:43,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47068 deadline: 1681995403394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,575 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/cc36db3460ca49d6a4976d6a790f8b5e 2023-04-20 12:55:43,581 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/a4e407d11b2e440a9c2679cab00ec5a0 2023-04-20 12:55:43,585 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/a4c07c0e94314eb8b28dfc257636743d as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/a4c07c0e94314eb8b28dfc257636743d 2023-04-20 12:55:43,588 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/a4c07c0e94314eb8b28dfc257636743d, entries=150, sequenceid=59, filesize=11.6 K 2023-04-20 12:55:43,588 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/cc36db3460ca49d6a4976d6a790f8b5e as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/cc36db3460ca49d6a4976d6a790f8b5e 2023-04-20 12:55:43,593 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/cc36db3460ca49d6a4976d6a790f8b5e, entries=150, sequenceid=59, filesize=11.6 K 2023-04-20 12:55:43,594 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/a4e407d11b2e440a9c2679cab00ec5a0 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/a4e407d11b2e440a9c2679cab00ec5a0 2023-04-20 12:55:43,597 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/a4e407d11b2e440a9c2679cab00ec5a0, entries=150, sequenceid=59, filesize=11.6 K 2023-04-20 12:55:43,598 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for bed4cc4c061ff4314d49ea7add6e3672 in 848ms, sequenceid=59, compaction requested=true 2023-04-20 12:55:43,598 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:55:43,598 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:43,598 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:43,598 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:43,598 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,598 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:43,598 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:43,599 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,599 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:43,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:43,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:43,599 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:43,599 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:43,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:43,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:43,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:43,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:43,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:43,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17c32531[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:43,600 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:43,600 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:43,600 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:43,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:43,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:43,601 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:43,601 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:43,601 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:43,601 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,601 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:43,604 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:43,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:43,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:43,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:43,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:43,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:43,606 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:43,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:43,606 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:43,607 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:43,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:43,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:43,607 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:43,607 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:43,607 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:43,607 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:43,608 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:43,615 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:43,615 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:43,615 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:43,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:43,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:43,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:43,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:43,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:43,615 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:43,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:43,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:43,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:43,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:43,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:43,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:43,616 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:43,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:43,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,618 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,618 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:43,618 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:43,618 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:43,618 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,618 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:43,618 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:43,618 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:43,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:43,618 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:43,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:43,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:43,618 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:43,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:43,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:43,619 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:43,619 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:43,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:43,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:43,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on bed4cc4c061ff4314d49ea7add6e3672 2023-04-20 12:55:43,624 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bed4cc4c061ff4314d49ea7add6e3672 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-20 12:55:43,624 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=A 2023-04-20 12:55:43,624 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:55:43,624 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=B 2023-04-20 12:55:43,624 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:55:43,624 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=C 2023-04-20 12:55:43,624 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:55:43,637 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:43,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47054 deadline: 1681995403632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,638 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:43,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54244 deadline: 1681995403633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,643 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:43,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54262 deadline: 1681995403637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,707 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:43,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54242 deadline: 1681995403698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,707 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:43,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47068 deadline: 1681995403699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,719 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:43,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:43,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:43,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:43,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:43,725 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:43,725 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:43,725 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:43,725 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:43,726 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:43,726 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:43,726 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,726 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:43,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:43,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:43,726 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,727 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:43,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:43,727 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:43,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:43,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:43,727 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:43,727 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:43,727 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:43,727 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:43,727 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:43,727 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:43,730 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:43,730 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:43,730 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:43,730 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:43,730 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:43,730 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:43,730 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:43,730 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:43,730 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,730 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:43,730 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:43,736 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,736 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:43,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:43,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:43,736 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:43,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:43,736 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:43,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:43,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:43,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:43,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:43,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63f7fa0f[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:43,737 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:43,737 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:43,740 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:43,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:43,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:43,740 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:43,740 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:43,740 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,740 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:43,740 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:43,741 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:43,741 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:43,741 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:43,741 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:43,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:43,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:43,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:43,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:43,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:43,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:43,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:43,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:43,743 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:43,743 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:43,743 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:43,743 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:43,743 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:43,744 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:43,744 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:43,744 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:43,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:43,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:43,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:43,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:43,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:43,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:43,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:43,744 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:43,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:43,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:43,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:43,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:43,745 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:43,746 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:43,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47054 deadline: 1681995403738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:43,746 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:43,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54244 deadline: 1681995403738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,747 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:43,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:43,747 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:43,747 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:43,747 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,747 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:43,747 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:43,747 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:43,747 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:43,747 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:43,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:43,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:43,748 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:43,747 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:43,747 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:43,747 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:43,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:43,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:43,753 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:43,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54262 deadline: 1681995403744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,848 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:43,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:43,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:43,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:43,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:43,850 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:43,850 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:43,851 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:43,851 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:43,851 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:43,851 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:43,851 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:43,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:43,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:43,852 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:43,852 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,852 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:43,852 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:43,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:43,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:43,852 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:43,853 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:43,853 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:43,853 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:43,853 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:43,853 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:43,853 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:43,853 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:43,853 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:43,853 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:43,853 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:43,853 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:43,853 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:43,853 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:43,853 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,853 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:43,853 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:43,854 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:43,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:43,854 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:43,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:43,854 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:43,854 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:43,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:43,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:43,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:43,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:43,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b0f7462[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:43,855 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:43,855 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:43,856 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:43,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:43,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:43,856 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:43,856 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:43,856 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:43,856 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,856 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:43,857 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,857 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:43,857 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:43,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:43,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:43,857 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:43,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:43,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:43,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:43,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:43,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:43,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:43,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:43,858 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:43,858 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:43,858 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:43,858 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:43,858 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:43,859 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:43,859 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:43,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:43,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:43,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:43,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:43,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:43,859 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:43,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:43,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:43,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:43,860 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:43,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:43,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:43,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:43,861 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:43,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:43,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,862 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,862 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:43,862 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:43,862 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:43,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:43,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:43,862 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:43,862 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:43,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:43,862 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:43,862 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:43,862 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:43,862 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:43,862 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:43,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:43,870 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:43,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:43,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:43,952 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:43,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47054 deadline: 1681995403947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,957 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:43,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54244 deadline: 1681995403948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,962 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:43,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54262 deadline: 1681995403954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,962 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:43,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:43,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:43,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:43,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:43,965 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:43,965 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:43,965 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:43,965 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:43,966 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:43,966 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:43,966 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:43,966 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:43,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:43,967 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,967 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:43,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:43,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:43,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:43,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:43,969 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:43,969 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:43,969 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:43,969 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:43,969 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:43,969 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:43,970 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:43,970 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:43,970 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:43,970 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:43,970 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:43,970 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:43,970 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:43,970 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:43,970 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,970 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:43,970 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:43,971 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:43,971 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:43,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:43,971 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:43,971 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:43,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:43,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:43,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:43,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:43,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:43,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a158a45[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:43,973 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:43,973 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:43,973 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:43,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:43,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:43,974 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:43,974 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:43,974 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:43,974 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,974 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:43,974 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:43,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:43,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:43,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:43,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:43,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,976 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:43,976 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:43,976 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:43,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:43,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:43,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:43,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:43,977 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:43,977 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:43,977 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:43,977 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:43,977 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:43,978 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:43,978 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:43,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:43,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:43,978 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:43,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:43,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:43,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:43,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:43,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:43,979 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:43,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:43,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:43,982 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:43,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:43,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:43,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:43,990 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,991 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:43,991 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:43,991 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:43,991 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:43,991 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:43,991 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:43,991 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:43,991 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:43,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:43,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:43,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:43,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:43,991 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:43,991 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:43,991 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:43,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:43,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:43,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:44,036 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/5634f84ae66941349a7597c5d60d3f86 2023-04-20 12:55:44,091 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:44,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:44,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:44,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:44,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:44,094 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:44,094 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:44,094 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:44,094 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:44,094 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:44,094 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:44,095 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:44,095 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:44,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:44,095 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,095 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:44,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:44,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:44,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:44,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:44,096 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:44,096 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:44,096 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:44,096 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:44,096 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:44,096 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:44,096 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:44,096 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:44,096 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:44,096 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:44,096 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:44,096 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:44,096 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:44,096 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:44,096 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,096 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:44,097 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:44,097 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:44,097 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:44,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:44,097 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:44,097 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:44,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:44,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:44,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:44,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:44,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:44,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54f1c69b[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:44,099 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:44,099 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:44,099 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:44,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:44,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:44,100 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:44,100 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:44,100 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:44,100 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,100 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:44,100 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:44,100 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:44,100 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:44,101 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:44,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:44,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:44,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:44,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:44,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:44,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:44,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:44,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:44,104 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:44,104 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:44,104 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:44,104 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:44,104 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:44,119 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:44,119 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:44,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:44,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:44,119 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:44,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:44,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:44,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:44,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:44,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:44,120 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:44,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:44,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:44,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:44,121 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:44,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:44,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:44,123 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:44,123 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:44,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:44,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:44,123 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:44,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:44,123 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:44,124 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:44,124 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,124 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:44,124 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:44,124 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:44,124 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:44,124 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:44,123 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:44,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:44,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:44,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:44,216 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:44,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54242 deadline: 1681995404210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,217 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:44,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47068 deadline: 1681995404210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,224 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:44,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:44,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:44,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:44,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:44,226 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:44,226 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:44,227 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:44,227 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:44,227 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:44,227 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:44,228 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:44,228 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:44,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:44,228 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,228 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:44,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:44,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:44,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:44,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:44,228 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:44,229 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:44,229 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:44,229 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:44,229 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:44,229 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:44,229 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:44,229 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:44,229 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:44,229 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:44,229 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:44,229 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:44,229 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:44,229 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:44,229 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,229 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:44,229 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:44,230 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:44,231 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:44,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:44,231 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:44,231 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:44,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:44,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:44,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:44,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:44,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:44,232 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:44,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@88a6988[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:44,232 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:44,232 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:44,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:44,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:44,232 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,233 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:44,233 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:44,233 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:44,233 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:44,233 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,233 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:44,233 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:44,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:44,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:44,233 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:44,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:44,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:44,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:44,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:44,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:44,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:44,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:44,235 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:44,235 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:44,235 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:44,235 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:44,235 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:44,236 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:44,236 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:44,236 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:44,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:44,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:44,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:44,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:44,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:44,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:44,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:44,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:44,236 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:44,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:44,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:44,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:44,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:44,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:44,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,238 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,239 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,239 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:44,239 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:44,239 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:44,239 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:44,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:44,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:44,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:44,239 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,239 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:44,239 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:44,239 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:44,239 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:44,240 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:44,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:44,239 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:44,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:44,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:44,265 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:44,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47054 deadline: 1681995404256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,265 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:44,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54244 deadline: 1681995404260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,273 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:44,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54262 deadline: 1681995404265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,340 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:44,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:44,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:44,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:44,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:44,343 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:44,343 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:44,343 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:44,343 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:44,344 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:44,344 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:44,344 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,344 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:44,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:44,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:44,344 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,344 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:44,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:44,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:44,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:44,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:44,345 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:44,345 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:44,345 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:44,345 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:44,345 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:44,345 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:44,346 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:44,346 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:44,346 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:44,346 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:44,346 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:44,346 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:44,346 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:44,346 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:44,346 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,346 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:44,346 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:44,347 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:44,347 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:44,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:44,347 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:44,347 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:44,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:44,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:44,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:44,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:44,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:44,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5876a09f[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:44,348 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:44,348 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:44,349 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,349 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:44,349 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:44,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:44,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:44,350 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:44,350 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:44,350 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:44,351 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,351 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:44,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:44,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:44,351 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:44,351 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:44,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:44,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:44,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:44,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:44,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:44,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:44,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:44,352 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:44,352 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:44,352 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:44,353 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:44,353 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:44,354 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:44,354 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:44,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:44,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:44,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:44,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:44,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:44,354 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:44,355 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:44,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:44,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:44,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:44,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:44,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:44,356 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:44,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:44,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:44,358 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,358 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:44,358 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:44,358 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,358 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:44,358 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:44,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:44,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:44,358 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:44,358 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:44,359 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:44,359 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:44,359 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:44,362 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:44,362 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:44,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:44,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:44,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:44,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:44,447 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/1a61932f7cc041dc8b1053bf00818529 2023-04-20 12:55:44,459 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:44,459 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/d598f73a75ee4296abcea320c44dd3c0 2023-04-20 12:55:44,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:44,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:44,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:44,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:44,463 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:44,463 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:44,463 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:44,463 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:44,463 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:44,463 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:44,464 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,464 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/5634f84ae66941349a7597c5d60d3f86 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/5634f84ae66941349a7597c5d60d3f86 2023-04-20 12:55:44,464 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,464 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:44,467 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:44,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:44,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:44,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:44,468 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:44,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:44,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:44,468 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:44,469 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:44,469 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:44,469 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:44,469 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:44,469 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:44,474 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/5634f84ae66941349a7597c5d60d3f86, entries=200, sequenceid=96, filesize=13.9 K 2023-04-20 12:55:44,474 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:44,474 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:44,474 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:44,474 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:44,474 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:44,474 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:44,474 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:44,474 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:44,474 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,474 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:44,474 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:44,476 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/1a61932f7cc041dc8b1053bf00818529 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/1a61932f7cc041dc8b1053bf00818529 2023-04-20 12:55:44,483 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/1a61932f7cc041dc8b1053bf00818529, entries=150, sequenceid=96, filesize=11.6 K 2023-04-20 12:55:44,484 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/d598f73a75ee4296abcea320c44dd3c0 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/d598f73a75ee4296abcea320c44dd3c0 2023-04-20 12:55:44,486 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:44,486 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:44,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:44,486 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:44,486 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:44,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:44,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:44,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:44,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:44,487 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/d598f73a75ee4296abcea320c44dd3c0, entries=150, sequenceid=96, filesize=11.6 K 2023-04-20 12:55:44,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:44,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ec74e76[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:44,488 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for bed4cc4c061ff4314d49ea7add6e3672 in 864ms, sequenceid=96, compaction requested=true 2023-04-20 12:55:44,488 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:55:44,488 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:44,488 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:55:44,488 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 12:55:44,488 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 12:55:44,488 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:55:44,488 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:44,489 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:55:44,489 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:44,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:44,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:44,489 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:44,489 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:44,490 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:44,490 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49884 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 12:55:44,490 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): bed4cc4c061ff4314d49ea7add6e3672/A is initiating minor compaction (all files) 2023-04-20 12:55:44,490 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 12:55:44,490 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bed4cc4c061ff4314d49ea7add6e3672/A in TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:44,490 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): bed4cc4c061ff4314d49ea7add6e3672/B is initiating minor compaction (all files) 2023-04-20 12:55:44,490 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/eccfdf12947f4871a6967f33fdbd6513, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/fa56b98987e74bc6957d962f0890b1ce, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/a4c07c0e94314eb8b28dfc257636743d, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/5634f84ae66941349a7597c5d60d3f86] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp, totalSize=48.7 K 2023-04-20 12:55:44,490 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bed4cc4c061ff4314d49ea7add6e3672/B in TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:44,490 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/9d68a7fcb1864ef3bdbc52c9b82459ec, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/7c0222ce61b343d9b7849e5c37eff46a, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/cc36db3460ca49d6a4976d6a790f8b5e, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/1a61932f7cc041dc8b1053bf00818529] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp, totalSize=46.4 K 2023-04-20 12:55:44,490 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting eccfdf12947f4871a6967f33fdbd6513, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1681995340853 2023-04-20 12:55:44,490 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 9d68a7fcb1864ef3bdbc52c9b82459ec, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1681995340853 2023-04-20 12:55:44,490 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting fa56b98987e74bc6957d962f0890b1ce, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1681995340877 2023-04-20 12:55:44,491 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 7c0222ce61b343d9b7849e5c37eff46a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1681995340877 2023-04-20 12:55:44,491 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting a4c07c0e94314eb8b28dfc257636743d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1681995341487 2023-04-20 12:55:44,491 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting cc36db3460ca49d6a4976d6a790f8b5e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1681995341487 2023-04-20 12:55:44,491 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 5634f84ae66941349a7597c5d60d3f86, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1681995343071 2023-04-20 12:55:44,491 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 1a61932f7cc041dc8b1053bf00818529, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1681995343071 2023-04-20 12:55:44,494 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,494 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:44,503 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): bed4cc4c061ff4314d49ea7add6e3672#B#compaction#391 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:55:44,504 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:44,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:44,504 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): bed4cc4c061ff4314d49ea7add6e3672#A#compaction#392 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:55:44,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:44,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:44,505 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:44,505 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:44,505 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:44,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:44,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:44,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:44,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:44,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:44,508 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:44,508 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:44,508 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:44,508 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:44,508 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:44,516 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:44,516 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:44,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:44,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:44,516 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:44,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:44,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:44,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:44,517 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,519 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,523 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,523 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:44,523 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:44,523 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:44,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:44,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:44,523 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,524 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:44,524 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:44,524 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:44,524 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:44,524 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:44,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:44,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:44,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:44,527 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:44,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:44,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:44,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:44,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:44,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:44,537 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:44,546 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/739606c1b6d9493786fad675e2d11c1e as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/739606c1b6d9493786fad675e2d11c1e 2023-04-20 12:55:44,551 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bed4cc4c061ff4314d49ea7add6e3672/B of bed4cc4c061ff4314d49ea7add6e3672 into 739606c1b6d9493786fad675e2d11c1e(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:55:44,551 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:55:44,551 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672., storeName=bed4cc4c061ff4314d49ea7add6e3672/B, priority=12, startTime=1681995344488; duration=0sec 2023-04-20 12:55:44,551 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:55:44,551 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 12:55:44,552 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 12:55:44,552 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): bed4cc4c061ff4314d49ea7add6e3672/C is initiating minor compaction (all files) 2023-04-20 12:55:44,553 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bed4cc4c061ff4314d49ea7add6e3672/C in TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:44,553 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/24a31044ddda451fbfe6116973c2411d, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/25d46b1616954b22ab16e3eb9cb8dd4f, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/a4e407d11b2e440a9c2679cab00ec5a0, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/d598f73a75ee4296abcea320c44dd3c0] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp, totalSize=46.4 K 2023-04-20 12:55:44,554 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 24a31044ddda451fbfe6116973c2411d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1681995340853 2023-04-20 12:55:44,554 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 25d46b1616954b22ab16e3eb9cb8dd4f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1681995340877 2023-04-20 12:55:44,555 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting a4e407d11b2e440a9c2679cab00ec5a0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1681995341487 2023-04-20 12:55:44,555 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting d598f73a75ee4296abcea320c44dd3c0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1681995343071 2023-04-20 12:55:44,564 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): bed4cc4c061ff4314d49ea7add6e3672#C#compaction#393 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:55:44,565 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/7cddb6ea1ec348e9a4861af368551fee as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/7cddb6ea1ec348e9a4861af368551fee 2023-04-20 12:55:44,570 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bed4cc4c061ff4314d49ea7add6e3672/A of bed4cc4c061ff4314d49ea7add6e3672 into 7cddb6ea1ec348e9a4861af368551fee(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:55:44,570 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:55:44,570 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672., storeName=bed4cc4c061ff4314d49ea7add6e3672/A, priority=12, startTime=1681995344488; duration=0sec 2023-04-20 12:55:44,570 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:55:44,586 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/98bb90f666dc43dbbbf4615088928fb7 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/98bb90f666dc43dbbbf4615088928fb7 2023-04-20 12:55:44,594 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bed4cc4c061ff4314d49ea7add6e3672/C of bed4cc4c061ff4314d49ea7add6e3672 into 98bb90f666dc43dbbbf4615088928fb7(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:55:44,594 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:55:44,594 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672., storeName=bed4cc4c061ff4314d49ea7add6e3672/C, priority=12, startTime=1681995344489; duration=0sec 2023-04-20 12:55:44,594 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:55:44,624 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:44,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:44,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:44,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:44,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:44,632 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:44,633 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:44,633 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:44,633 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:44,633 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:44,633 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:44,634 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:44,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:44,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:44,634 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:44,634 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,634 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:44,634 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:44,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:44,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:44,635 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:44,638 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:44,638 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:44,638 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:44,638 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:44,638 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:44,649 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:44,649 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:44,649 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:44,649 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:44,649 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bed4cc4c061ff4314d49ea7add6e3672 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-20 12:55:44,649 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=A 2023-04-20 12:55:44,649 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:44,650 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=B 2023-04-20 12:55:44,650 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:44,650 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=C 2023-04-20 12:55:44,650 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:44,678 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/cfaf857cfd9f4d1484cad268026e6991 2023-04-20 12:55:44,698 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/43a0cfa0def049c392188bdd28df2005 2023-04-20 12:55:44,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on bed4cc4c061ff4314d49ea7add6e3672 2023-04-20 12:55:44,770 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:44,803 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681995344791, max timestamp=1681995344800Num uniques 859; 2023-04-20 12:55:44,803 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 12:55:44,803 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681995344791, max timestamp=1681995344800Num uniques 859; 2023-04-20 12:55:44,803 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 12:55:44,803 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681995344791, max timestamp=1681995344800Num uniques 859; 2023-04-20 12:55:44,803 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 12:55:44,860 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:44,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47054 deadline: 1681995404852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,865 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:44,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54262 deadline: 1681995404856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,868 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:44,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54244 deadline: 1681995404861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,985 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:44,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47054 deadline: 1681995404961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,985 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:44,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54262 deadline: 1681995404983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:44,985 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:44,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54244 deadline: 1681995404984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,110 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/7d67ced317a9483580ac5c683cfc9ab5 2023-04-20 12:55:45,114 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/cfaf857cfd9f4d1484cad268026e6991 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/cfaf857cfd9f4d1484cad268026e6991 2023-04-20 12:55:45,118 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/cfaf857cfd9f4d1484cad268026e6991, entries=150, sequenceid=108, filesize=11.6 K 2023-04-20 12:55:45,119 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/43a0cfa0def049c392188bdd28df2005 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/43a0cfa0def049c392188bdd28df2005 2023-04-20 12:55:45,136 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/43a0cfa0def049c392188bdd28df2005, entries=150, sequenceid=108, filesize=11.6 K 2023-04-20 12:55:45,137 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/7d67ced317a9483580ac5c683cfc9ab5 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/7d67ced317a9483580ac5c683cfc9ab5 2023-04-20 12:55:45,156 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/7d67ced317a9483580ac5c683cfc9ab5, entries=150, sequenceid=108, filesize=11.6 K 2023-04-20 12:55:45,157 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for bed4cc4c061ff4314d49ea7add6e3672 in 508ms, sequenceid=108, compaction requested=false 2023-04-20 12:55:45,157 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:55:45,157 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:45,157 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:45,157 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:45,157 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,158 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:45,158 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:45,159 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:45,159 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:45,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:45,159 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:45,159 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:45,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:45,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:45,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:45,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:45,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:45,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74d8802b[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:45,161 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:45,161 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:45,161 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:45,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:45,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:45,161 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:45,161 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:45,162 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:45,162 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,162 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:45,162 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,162 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:45,162 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:45,162 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:45,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:45,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:45,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:45,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:45,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:45,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:45,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:45,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:45,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:45,165 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:45,165 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:45,165 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:45,165 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:45,165 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:45,166 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,166 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:45,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:45,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,166 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:45,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:45,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:45,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:45,167 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:45,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:45,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:45,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:45,168 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:45,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,170 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:45,170 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:45,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:45,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:45,170 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,171 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:45,171 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:45,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:45,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:45,171 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:45,171 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,171 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:45,171 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:45,171 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,171 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:45,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:45,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:45,171 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:45,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681995344851, max timestamp=1681995345190Num uniques 1068; 2023-04-20 12:55:45,192 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-20 12:55:45,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on bed4cc4c061ff4314d49ea7add6e3672 2023-04-20 12:55:45,192 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bed4cc4c061ff4314d49ea7add6e3672 3/3 column families, dataSize=107.34 KB heapSize=302.36 KB 2023-04-20 12:55:45,193 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=A 2023-04-20 12:55:45,193 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:55:45,193 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=B 2023-04-20 12:55:45,193 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-20 12:55:45,194 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=C 2023-04-20 12:55:45,194 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-20 12:55:45,232 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/4f83e69989314dbcb8a877d17d453191 2023-04-20 12:55:45,234 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:45,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54244 deadline: 1681995405222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,234 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:45,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54262 deadline: 1681995405222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,235 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:45,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54242 deadline: 1681995405225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,235 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:45,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47068 deadline: 1681995405227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,246 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:45,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47054 deadline: 1681995405235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,256 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/40a86b744558418da08bb3a6bec972c4 2023-04-20 12:55:45,271 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:45,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:45,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:45,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:45,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:45,283 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:45,283 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:45,283 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:45,283 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:45,283 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,283 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:45,284 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:45,284 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:45,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:45,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:45,285 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,285 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:45,285 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:45,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:45,285 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:45,298 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:45,298 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:45,298 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:45,298 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:45,298 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:45,298 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/98de14f7e898410aa6aaceaa998bfc6d 2023-04-20 12:55:45,302 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:45,302 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:45,302 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:45,302 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:45,302 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:45,302 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:45,302 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:45,302 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/4f83e69989314dbcb8a877d17d453191 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/4f83e69989314dbcb8a877d17d453191 2023-04-20 12:55:45,302 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:45,303 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,303 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:45,303 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:45,303 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,303 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:45,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:45,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:45,304 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:45,304 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:45,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:45,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:45,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:45,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:45,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:45,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f456111[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:45,306 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:45,306 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:45,307 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,307 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:45,309 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:45,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:45,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:45,309 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:45,309 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:45,309 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:45,310 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:45,310 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:45,310 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:45,311 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:45,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:45,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:45,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:45,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:45,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:45,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:45,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:45,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:45,312 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:45,312 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:45,312 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:45,312 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:45,312 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:45,313 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:45,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:45,313 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,313 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:45,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:45,314 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:45,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:45,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:45,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:45,315 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/4f83e69989314dbcb8a877d17d453191, entries=150, sequenceid=146, filesize=11.8 K 2023-04-20 12:55:45,315 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,315 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/40a86b744558418da08bb3a6bec972c4 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/40a86b744558418da08bb3a6bec972c4 2023-04-20 12:55:45,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:45,317 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,317 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:45,317 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:45,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:45,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:45,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:45,317 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:45,317 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:45,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:45,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:45,317 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:45,317 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,317 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:45,317 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:45,317 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:45,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:45,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:45,317 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:45,317 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:45,319 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/40a86b744558418da08bb3a6bec972c4, entries=150, sequenceid=146, filesize=11.8 K 2023-04-20 12:55:45,319 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/98de14f7e898410aa6aaceaa998bfc6d as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/98de14f7e898410aa6aaceaa998bfc6d 2023-04-20 12:55:45,322 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/98de14f7e898410aa6aaceaa998bfc6d, entries=150, sequenceid=146, filesize=11.8 K 2023-04-20 12:55:45,322 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~302.31 KB/309568, currentSize=73.80 KB/75570 for bed4cc4c061ff4314d49ea7add6e3672 in 130ms, sequenceid=146, compaction requested=true 2023-04-20 12:55:45,322 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:55:45,322 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:55:45,322 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:55:45,322 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:55:45,322 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:55:45,323 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:55:45,323 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35946 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:55:45,323 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35946 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:55:45,323 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): bed4cc4c061ff4314d49ea7add6e3672/A is initiating minor compaction (all files) 2023-04-20 12:55:45,323 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): bed4cc4c061ff4314d49ea7add6e3672/B is initiating minor compaction (all files) 2023-04-20 12:55:45,323 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bed4cc4c061ff4314d49ea7add6e3672/A in TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:45,323 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bed4cc4c061ff4314d49ea7add6e3672/B in TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:45,323 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/7cddb6ea1ec348e9a4861af368551fee, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/cfaf857cfd9f4d1484cad268026e6991, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/4f83e69989314dbcb8a877d17d453191] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp, totalSize=35.1 K 2023-04-20 12:55:45,323 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/739606c1b6d9493786fad675e2d11c1e, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/43a0cfa0def049c392188bdd28df2005, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/40a86b744558418da08bb3a6bec972c4] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp, totalSize=35.1 K 2023-04-20 12:55:45,324 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 7cddb6ea1ec348e9a4861af368551fee, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1681995343071 2023-04-20 12:55:45,324 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 739606c1b6d9493786fad675e2d11c1e, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1681995343071 2023-04-20 12:55:45,324 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting cfaf857cfd9f4d1484cad268026e6991, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1681995343628 2023-04-20 12:55:45,324 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 43a0cfa0def049c392188bdd28df2005, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1681995343628 2023-04-20 12:55:45,324 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 4f83e69989314dbcb8a877d17d453191, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1681995344851 2023-04-20 12:55:45,325 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 40a86b744558418da08bb3a6bec972c4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1681995344851 2023-04-20 12:55:45,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on bed4cc4c061ff4314d49ea7add6e3672 2023-04-20 12:55:45,338 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bed4cc4c061ff4314d49ea7add6e3672 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-20 12:55:45,339 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): bed4cc4c061ff4314d49ea7add6e3672#A#compaction#400 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:55:45,339 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=A 2023-04-20 12:55:45,339 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:45,339 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=B 2023-04-20 12:55:45,339 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:45,339 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=C 2023-04-20 12:55:45,339 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:45,339 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): bed4cc4c061ff4314d49ea7add6e3672#B#compaction#401 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:55:45,370 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/18ea54d6fc2e4aa3a28a99e7083cd327 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/18ea54d6fc2e4aa3a28a99e7083cd327 2023-04-20 12:55:45,374 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bed4cc4c061ff4314d49ea7add6e3672/B of bed4cc4c061ff4314d49ea7add6e3672 into 18ea54d6fc2e4aa3a28a99e7083cd327(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:55:45,374 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:55:45,374 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672., storeName=bed4cc4c061ff4314d49ea7add6e3672/B, priority=13, startTime=1681995345322; duration=0sec 2023-04-20 12:55:45,374 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:55:45,374 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:55:45,376 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35946 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:55:45,376 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): bed4cc4c061ff4314d49ea7add6e3672/C is initiating minor compaction (all files) 2023-04-20 12:55:45,377 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bed4cc4c061ff4314d49ea7add6e3672/C in TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:45,377 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/98bb90f666dc43dbbbf4615088928fb7, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/7d67ced317a9483580ac5c683cfc9ab5, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/98de14f7e898410aa6aaceaa998bfc6d] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp, totalSize=35.1 K 2023-04-20 12:55:45,377 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 98bb90f666dc43dbbbf4615088928fb7, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1681995343071 2023-04-20 12:55:45,377 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 7d67ced317a9483580ac5c683cfc9ab5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1681995343628 2023-04-20 12:55:45,377 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 98de14f7e898410aa6aaceaa998bfc6d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1681995344851 2023-04-20 12:55:45,386 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): bed4cc4c061ff4314d49ea7add6e3672#C#compaction#403 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:55:45,391 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:45,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47054 deadline: 1681995405384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,391 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:45,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54262 deadline: 1681995405384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,392 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:45,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54244 deadline: 1681995405385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,418 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:45,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:45,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:45,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:45,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:45,420 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:45,420 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:45,420 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:45,420 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:45,420 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,420 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:45,422 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:45,422 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:45,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:45,422 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,422 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:45,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:45,423 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:45,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:45,423 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:45,423 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:45,423 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:45,423 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:45,423 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:45,423 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:45,424 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:45,424 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:45,424 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:45,424 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:45,424 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:45,424 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:45,424 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:45,424 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:45,424 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,424 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:45,424 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:45,425 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,425 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:45,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:45,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:45,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:45,425 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:45,425 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:45,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:45,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:45,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:45,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:45,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42da4e3e[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:45,426 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:45,426 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:45,427 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:45,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:45,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:45,427 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:45,427 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:45,427 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:45,427 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,427 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:45,428 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,428 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:45,428 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:45,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:45,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:45,428 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:45,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:45,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:45,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:45,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:45,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:45,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:45,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:45,430 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:45,430 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:45,430 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:45,430 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:45,430 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:45,431 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,431 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:45,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:45,431 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:45,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:45,431 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:45,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:45,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:45,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:45,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:45,432 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:45,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:45,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,434 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,434 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:45,434 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:45,434 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:45,434 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:45,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:45,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:45,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:45,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:45,435 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:45,434 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,435 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:45,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:45,435 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:45,435 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:45,435 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:45,435 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:45,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:45,498 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:45,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47054 deadline: 1681995405492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,498 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:45,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54262 deadline: 1681995405492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,498 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:45,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54244 deadline: 1681995405492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,535 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:45,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:45,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:45,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:45,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:45,539 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:45,539 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:45,539 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:45,539 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:45,539 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,539 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:45,540 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:45,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:45,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:45,540 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:45,540 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,540 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:45,541 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:45,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:45,541 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:45,541 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:45,541 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:45,541 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:45,541 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:45,541 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:45,541 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:45,542 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:45,542 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:45,542 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:45,542 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:45,542 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:45,542 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:45,542 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:45,542 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,542 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:45,542 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:45,543 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:45,543 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,543 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:45,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:45,543 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:45,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:45,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:45,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:45,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:45,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:45,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:45,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56fa1ced[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:45,546 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:45,546 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:45,546 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:45,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:45,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:45,547 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:45,547 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,547 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:45,547 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:45,547 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:45,548 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:45,548 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:45,548 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:45,548 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:45,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:45,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:45,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:45,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:45,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:45,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:45,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:45,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:45,551 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:45,551 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:45,551 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:45,551 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:45,551 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:45,552 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,552 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,552 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:45,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:45,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:45,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:45,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:45,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:45,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,553 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:45,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:45,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:45,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:45,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:45,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,558 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,560 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:45,560 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:45,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:45,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:45,560 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,560 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:45,561 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:45,561 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,561 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:45,561 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:45,561 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:45,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:45,566 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:45,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:45,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:45,566 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:45,566 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:45,666 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:45,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:45,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:45,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:45,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:45,674 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:45,674 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:45,674 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:45,674 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:45,675 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,675 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:45,676 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:45,677 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:45,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:45,677 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,677 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:45,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:45,677 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:45,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:45,678 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:45,678 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:45,678 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:45,678 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:45,678 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:45,678 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:45,678 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:45,678 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:45,678 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:45,678 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:45,678 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:45,678 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:45,679 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:45,679 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:45,679 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,679 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:45,679 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:45,680 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,680 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:45,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:45,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:45,680 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:45,680 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:45,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:45,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:45,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:45,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:45,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:45,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@669317c1[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:45,685 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:45,685 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:45,686 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:45,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:45,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:45,686 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:45,686 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:45,686 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:45,687 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:45,687 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:45,687 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:45,687 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,688 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:45,688 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:45,688 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:45,688 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:45,688 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:45,688 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:45,688 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:45,690 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,690 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:45,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:45,690 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:45,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:45,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:45,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:45,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:45,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,691 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:45,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:45,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,692 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:45,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:45,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,694 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:45,694 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:45,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:45,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:45,694 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,694 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:45,694 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:45,694 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,694 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:45,694 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:45,694 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:45,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:45,696 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:45,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:45,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:45,696 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:45,696 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:45,704 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:45,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47054 deadline: 1681995405699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,704 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:45,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54262 deadline: 1681995405700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,706 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:45,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54244 deadline: 1681995405700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,761 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/cc265e856a6c4f4a86619cb3c976e3e5 2023-04-20 12:55:45,784 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/be45eb10900f48ac9dafcd62954a767b as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/be45eb10900f48ac9dafcd62954a767b 2023-04-20 12:55:45,792 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bed4cc4c061ff4314d49ea7add6e3672/A of bed4cc4c061ff4314d49ea7add6e3672 into be45eb10900f48ac9dafcd62954a767b(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:55:45,792 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:55:45,792 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672., storeName=bed4cc4c061ff4314d49ea7add6e3672/A, priority=13, startTime=1681995345322; duration=0sec 2023-04-20 12:55:45,793 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:55:45,796 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:45,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:45,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:45,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:45,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:45,803 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:45,803 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:45,803 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:45,803 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:45,806 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,806 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:45,807 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:45,807 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:45,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:45,807 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,807 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:45,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:45,808 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:45,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:45,808 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:45,808 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:45,808 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:45,808 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:45,808 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:45,808 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:45,809 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:45,809 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:45,809 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:45,809 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/f45f6f12d5b049b1b6caf833ee52a3ab as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/f45f6f12d5b049b1b6caf833ee52a3ab 2023-04-20 12:55:45,809 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:45,809 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:45,809 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:45,809 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:45,809 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:45,809 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,809 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:45,809 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:45,811 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:45,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:45,811 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:45,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:45,812 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:45,812 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:45,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:45,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:45,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:45,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:45,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62daafeb[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:45,814 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:45,815 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:45,815 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bed4cc4c061ff4314d49ea7add6e3672/C of bed4cc4c061ff4314d49ea7add6e3672 into f45f6f12d5b049b1b6caf833ee52a3ab(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:55:45,815 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:55:45,815 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672., storeName=bed4cc4c061ff4314d49ea7add6e3672/C, priority=13, startTime=1681995345323; duration=0sec 2023-04-20 12:55:45,815 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:55:45,823 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/bdc728ee7c904c85a926570806640adf 2023-04-20 12:55:45,823 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:45,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:45,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:45,823 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:45,823 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:45,823 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:45,824 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,824 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:45,824 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,824 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:45,824 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:45,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:45,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:45,824 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:45,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:45,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:45,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:45,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:45,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:45,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:45,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:45,827 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:45,827 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:45,827 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:45,827 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:45,827 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:45,829 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,829 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:45,829 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:45,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:45,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:45,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:45,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:45,831 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:45,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:45,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:45,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:45,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:45,832 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,840 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/0ed6702a5dc94f42a8c81f7926d8f219 2023-04-20 12:55:45,844 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,844 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:45,844 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:45,844 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,844 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:45,844 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:45,844 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,845 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:45,845 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:45,845 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:45,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:45,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:45,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:45,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:45,845 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:45,845 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:45,846 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/cc265e856a6c4f4a86619cb3c976e3e5 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/cc265e856a6c4f4a86619cb3c976e3e5 2023-04-20 12:55:45,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:45,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:45,850 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/cc265e856a6c4f4a86619cb3c976e3e5, entries=250, sequenceid=162, filesize=16.4 K 2023-04-20 12:55:45,851 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/bdc728ee7c904c85a926570806640adf as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/bdc728ee7c904c85a926570806640adf 2023-04-20 12:55:45,855 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/bdc728ee7c904c85a926570806640adf, entries=150, sequenceid=162, filesize=11.8 K 2023-04-20 12:55:45,855 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/0ed6702a5dc94f42a8c81f7926d8f219 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/0ed6702a5dc94f42a8c81f7926d8f219 2023-04-20 12:55:45,860 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/0ed6702a5dc94f42a8c81f7926d8f219, entries=150, sequenceid=162, filesize=11.8 K 2023-04-20 12:55:45,861 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for bed4cc4c061ff4314d49ea7add6e3672 in 523ms, sequenceid=162, compaction requested=false 2023-04-20 12:55:45,861 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:55:45,946 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:45,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:45,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:45,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:45,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:45,954 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:45,954 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:45,954 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:45,954 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:45,954 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,954 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:45,955 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:45,955 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:45,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:45,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:45,956 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:45,956 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:45,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:45,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:45,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:45,958 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:45,959 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:45,959 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:45,959 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:45,959 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:45,959 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:45,962 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:45,962 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:45,962 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:45,962 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:45,962 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bed4cc4c061ff4314d49ea7add6e3672 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-20 12:55:45,962 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=A 2023-04-20 12:55:45,962 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:45,962 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=B 2023-04-20 12:55:45,962 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:45,963 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=C 2023-04-20 12:55:45,963 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:46,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on bed4cc4c061ff4314d49ea7add6e3672 2023-04-20 12:55:46,010 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:46,034 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:46,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47054 deadline: 1681995406028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:46,034 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:46,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54244 deadline: 1681995406029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:46,035 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:46,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54262 deadline: 1681995406030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:46,141 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:46,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54244 deadline: 1681995406135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:46,141 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:46,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47054 deadline: 1681995406135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:46,141 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:46,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54262 deadline: 1681995406135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:46,348 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:46,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54262 deadline: 1681995406344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:46,348 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:46,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47054 deadline: 1681995406345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:46,348 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:46,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54244 deadline: 1681995406347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:46,379 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/4caa68eee05b480598992e38f1b18389 2023-04-20 12:55:46,657 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:46,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54244 deadline: 1681995406649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:46,657 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:46,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54262 deadline: 1681995406650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:46,658 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:46,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47054 deadline: 1681995406651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:46,790 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/4d21231fed0a4e63b06b46e442c0cbb2 2023-04-20 12:55:46,812 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/b87fc7f34f124cbc9138305a3b36c026 2023-04-20 12:55:46,817 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/4caa68eee05b480598992e38f1b18389 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/4caa68eee05b480598992e38f1b18389 2023-04-20 12:55:46,820 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/4caa68eee05b480598992e38f1b18389, entries=150, sequenceid=181, filesize=11.8 K 2023-04-20 12:55:46,821 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/4d21231fed0a4e63b06b46e442c0cbb2 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/4d21231fed0a4e63b06b46e442c0cbb2 2023-04-20 12:55:46,827 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/4d21231fed0a4e63b06b46e442c0cbb2, entries=150, sequenceid=181, filesize=11.8 K 2023-04-20 12:55:46,828 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/b87fc7f34f124cbc9138305a3b36c026 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/b87fc7f34f124cbc9138305a3b36c026 2023-04-20 12:55:46,833 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/b87fc7f34f124cbc9138305a3b36c026, entries=150, sequenceid=181, filesize=11.8 K 2023-04-20 12:55:46,834 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for bed4cc4c061ff4314d49ea7add6e3672 in 872ms, sequenceid=181, compaction requested=true 2023-04-20 12:55:46,834 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:55:46,834 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:46,834 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:46,834 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:46,834 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:46,838 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:46,838 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:46,839 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:46,839 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:46,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:46,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:46,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:46,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:46,839 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:46,839 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:46,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:46,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:46,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:46,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:46,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:46,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7261a630[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:46,844 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:46,844 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:46,848 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:46,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:46,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:46,848 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:46,848 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:46,848 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:46,848 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:46,848 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:46,850 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:46,850 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:46,850 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:46,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:46,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:46,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:46,850 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:46,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:46,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:46,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:46,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:46,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:46,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:46,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:46,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:46,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:46,852 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:46,853 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:46,853 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:46,853 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:46,853 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:46,854 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:46,854 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:46,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:46,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:46,854 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:46,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:46,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:46,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:46,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:46,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:46,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:46,855 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:46,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:46,856 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:46,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:46,857 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:46,857 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:46,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:46,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:46,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:46,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:46,858 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:46,858 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:46,858 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:46,858 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:46,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:46,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:46,858 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:46,858 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:46,858 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:46,858 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:46,858 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:46,858 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:46,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:46,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:46,859 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:46,958 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:46,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:46,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:46,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:46,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:46,961 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:46,961 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:46,961 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:46,961 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:46,962 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:46,962 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:46,962 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:46,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:46,962 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:46,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:46,963 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:46,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:46,963 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:46,963 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:46,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:46,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:46,964 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:46,964 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:46,964 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:46,964 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:46,964 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:46,964 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:46,970 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:46,970 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:46,970 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:46,970 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:46,970 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bed4cc4c061ff4314d49ea7add6e3672 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-20 12:55:46,970 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=A 2023-04-20 12:55:46,970 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:46,970 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=B 2023-04-20 12:55:46,970 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:46,970 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=C 2023-04-20 12:55:46,970 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:47,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on bed4cc4c061ff4314d49ea7add6e3672 2023-04-20 12:55:47,168 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:47,219 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:47,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47054 deadline: 1681995407211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,220 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:47,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54262 deadline: 1681995407212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,225 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:47,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54244 deadline: 1681995407220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,249 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:47,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47068 deadline: 1681995407242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,251 DEBUG [Thread-3087] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4167 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672., hostname=jenkins-hbase20.apache.org,36277,1681995213947, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-20 12:55:47,260 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:47,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54242 deadline: 1681995407252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,264 DEBUG [Thread-3079] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4184 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672., hostname=jenkins-hbase20.apache.org,36277,1681995213947, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-20 12:55:47,325 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:47,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54262 deadline: 1681995407321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,326 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:47,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47054 deadline: 1681995407323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,336 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:47,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54244 deadline: 1681995407328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,384 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/d39746c2ae814a0d83b069e7abcdae2d 2023-04-20 12:55:47,408 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/6358f051bdca4ca5baa2d86a1cbc594f 2023-04-20 12:55:47,428 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/890060bf910542a8ad8f67e0e0c170b6 2023-04-20 12:55:47,431 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/d39746c2ae814a0d83b069e7abcdae2d as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/d39746c2ae814a0d83b069e7abcdae2d 2023-04-20 12:55:47,436 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/d39746c2ae814a0d83b069e7abcdae2d, entries=150, sequenceid=197, filesize=11.8 K 2023-04-20 12:55:47,437 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/6358f051bdca4ca5baa2d86a1cbc594f as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/6358f051bdca4ca5baa2d86a1cbc594f 2023-04-20 12:55:47,443 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/6358f051bdca4ca5baa2d86a1cbc594f, entries=150, sequenceid=197, filesize=11.8 K 2023-04-20 12:55:47,445 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/890060bf910542a8ad8f67e0e0c170b6 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/890060bf910542a8ad8f67e0e0c170b6 2023-04-20 12:55:47,448 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/890060bf910542a8ad8f67e0e0c170b6, entries=150, sequenceid=197, filesize=11.8 K 2023-04-20 12:55:47,448 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for bed4cc4c061ff4314d49ea7add6e3672 in 478ms, sequenceid=197, compaction requested=true 2023-04-20 12:55:47,448 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:55:47,448 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:47,448 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:47,449 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:47,449 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,449 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:47,449 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:47,449 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:47,449 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:47,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:47,450 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:47,450 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:47,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:47,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:47,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:47,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:47,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:47,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74fad1e5[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:47,451 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:47,451 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:47,451 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:47,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:47,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:47,452 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,452 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:47,452 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:47,452 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:47,452 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:47,452 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,452 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:47,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:47,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:47,452 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:47,452 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:47,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:47,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:47,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:47,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:47,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:47,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:47,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:47,455 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:47,455 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:47,455 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:47,455 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:47,455 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:47,456 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:47,456 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:47,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:47,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:47,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:47,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:47,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:47,456 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:47,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:47,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:47,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:47,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:47,457 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:47,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:47,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:47,458 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:47,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:47,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,460 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:47,460 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,460 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:47,460 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:47,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:47,460 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:47,460 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:47,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:47,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:47,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:47,460 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:47,460 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,460 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:47,460 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:47,460 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:47,461 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:47,461 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:47,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:47,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:47,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on bed4cc4c061ff4314d49ea7add6e3672 2023-04-20 12:55:47,531 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bed4cc4c061ff4314d49ea7add6e3672 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-20 12:55:47,532 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=A 2023-04-20 12:55:47,532 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:47,532 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=B 2023-04-20 12:55:47,532 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:47,532 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=C 2023-04-20 12:55:47,532 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:47,561 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=215 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/b66fabb8f0f440a393e596520960776e 2023-04-20 12:55:47,561 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:47,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:47,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:47,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:47,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:47,570 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:47,570 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:47,570 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:47,570 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:47,571 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:47,571 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:47,573 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:47,573 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,573 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,573 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:47,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:47,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:47,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:47,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:47,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:47,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:47,574 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:47,575 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:47,575 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:47,575 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:47,575 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:47,575 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:47,575 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:47,575 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:47,575 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:47,575 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:47,575 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:47,575 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:47,575 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:47,575 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:47,576 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,576 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:47,576 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:47,576 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:47,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:47,577 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:47,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:47,577 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:47,577 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:47,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:47,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:47,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:47,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:47,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@aa9eda4[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:47,584 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:47,584 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:47,585 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:47,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54244 deadline: 1681995407568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,585 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:47,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:47,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:47,585 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,586 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:47,586 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:47,586 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:47,586 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:47,586 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=215 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/662f2d05bf45400289316be46c6d67e2 2023-04-20 12:55:47,587 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,587 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:47,587 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:47,587 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:47,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:47,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:47,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:47,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:47,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:47,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:47,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:47,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:47,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:47,591 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:47,591 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:47,591 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:47,591 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:47,591 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:47,593 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:47,593 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:47,593 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:47,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:47,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:47,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:47,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:47,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:47,593 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:47,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54262 deadline: 1681995407578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:47,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:47,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:47,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:47,594 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:47,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:47,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:47,595 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:47,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:47,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,596 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:47,596 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:47,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:47,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:47,596 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,597 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:47,597 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:47,597 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,597 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:47,597 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:47,597 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:47,597 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:47,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:47,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:47,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:47,599 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:47,599 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:47,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:47,599 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:47,599 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:47,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47054 deadline: 1681995407585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,693 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:47,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54244 deadline: 1681995407686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,699 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:47,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:47,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:47,705 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:47,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54262 deadline: 1681995407694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,707 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:47,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47054 deadline: 1681995407700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:47,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:47,710 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:47,710 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:47,710 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:47,710 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:47,710 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:47,711 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:47,711 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:47,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:47,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:47,711 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:47,712 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,712 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:47,713 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:47,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:47,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:47,714 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:47,718 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:47,718 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:47,718 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:47,718 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:47,718 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:47,718 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:47,718 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:47,718 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:47,718 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:47,718 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:47,718 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:47,718 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:47,719 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:47,719 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,719 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:47,719 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:47,731 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:47,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:47,731 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:47,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:47,733 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:47,733 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:47,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:47,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:47,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:47,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:47,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d3bbeed[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:47,734 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:47,734 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:47,736 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:47,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:47,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:47,736 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:47,736 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:47,736 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:47,736 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,736 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:47,737 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:47,737 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:47,737 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:47,737 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:47,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:47,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:47,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:47,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:47,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:47,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:47,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:47,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:47,739 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:47,739 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:47,739 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:47,739 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:47,739 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:47,740 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:47,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:47,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:47,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:47,740 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:47,740 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:47,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:47,740 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:47,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:47,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:47,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:47,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:47,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:47,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:47,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:47,742 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:47,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:47,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,744 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,744 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:47,744 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:47,744 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,744 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:47,744 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:47,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:47,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:47,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:47,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:47,744 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:47,744 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:47,744 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:47,744 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:47,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:47,744 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:47,744 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:47,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:47,744 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:47,844 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:47,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:47,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:47,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:47,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:47,847 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:47,847 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:47,847 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:47,847 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:47,847 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:47,847 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:47,848 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,848 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:47,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:47,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:47,848 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,848 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:47,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:47,849 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:47,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:47,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:47,849 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:47,849 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:47,849 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:47,849 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:47,849 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:47,849 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:47,850 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:47,850 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:47,850 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:47,850 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:47,850 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:47,850 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:47,850 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:47,850 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:47,850 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,850 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:47,850 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:47,851 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,851 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:47,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:47,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:47,851 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:47,851 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:47,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:47,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:47,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:47,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:47,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:47,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4294cf5c[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:47,852 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:47,853 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:47,853 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:47,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:47,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:47,853 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:47,853 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:47,853 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:47,854 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,854 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:47,854 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,854 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:47,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:47,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:47,854 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:47,854 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:47,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:47,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:47,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:47,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:47,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:47,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:47,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:47,856 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:47,856 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:47,856 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:47,856 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:47,856 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:47,857 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:47,857 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:47,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:47,857 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:47,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:47,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:47,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:47,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:47,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:47,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:47,857 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:47,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:47,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:47,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:47,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:47,858 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:47,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:47,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,860 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:47,860 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:47,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:47,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:47,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:47,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:47,860 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,860 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:47,860 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:47,860 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:47,860 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:47,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:47,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:47,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:47,860 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:47,860 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,860 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:47,860 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:47,860 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:47,898 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:47,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54244 deadline: 1681995407895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,910 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:47,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54262 deadline: 1681995407907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,914 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:47,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47054 deadline: 1681995407910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,960 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:47,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:47,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:47,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:47,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:47,963 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:47,963 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:47,963 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:47,963 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:47,963 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:47,963 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:47,964 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:47,964 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:47,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:47,964 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,964 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:47,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:47,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:47,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:47,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:47,965 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:47,965 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:47,965 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:47,965 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:47,965 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:47,965 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:47,965 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:47,965 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:47,965 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:47,965 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:47,965 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:47,965 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:47,965 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:47,965 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:47,965 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,966 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:47,966 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:47,966 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,966 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:47,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:47,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:47,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:47,967 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:47,967 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:47,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:47,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:47,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:47,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:47,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60435b12[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:47,967 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:47,967 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:47,968 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:47,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:47,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:47,968 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:47,968 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:47,968 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:47,968 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,968 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:47,969 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,969 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:47,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:47,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:47,969 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:47,969 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:47,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:47,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:47,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:47,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:47,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:47,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:47,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:47,970 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:47,971 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:47,971 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:47,971 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:47,971 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:47,972 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:47,972 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:47,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:47,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:47,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:47,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:47,973 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,973 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:47,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:47,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:47,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:47,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:47,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:47,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:47,974 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:47,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:47,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:47,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:47,975 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,975 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:47,975 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:47,975 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:47,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:47,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:47,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:47,975 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:47,975 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:47,975 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:47,975 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:47,976 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:47,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:47,975 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:47,976 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:47,976 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:47,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:47,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:47,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:47,996 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=215 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/a90d582e84094defbc91492388725ebf 2023-04-20 12:55:47,999 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/b66fabb8f0f440a393e596520960776e as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/b66fabb8f0f440a393e596520960776e 2023-04-20 12:55:48,003 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/b66fabb8f0f440a393e596520960776e, entries=250, sequenceid=215, filesize=16.4 K 2023-04-20 12:55:48,003 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/662f2d05bf45400289316be46c6d67e2 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/662f2d05bf45400289316be46c6d67e2 2023-04-20 12:55:48,007 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/662f2d05bf45400289316be46c6d67e2, entries=150, sequenceid=215, filesize=11.8 K 2023-04-20 12:55:48,007 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/a90d582e84094defbc91492388725ebf as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/a90d582e84094defbc91492388725ebf 2023-04-20 12:55:48,018 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/a90d582e84094defbc91492388725ebf, entries=150, sequenceid=215, filesize=11.8 K 2023-04-20 12:55:48,020 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for bed4cc4c061ff4314d49ea7add6e3672 in 489ms, sequenceid=215, compaction requested=true 2023-04-20 12:55:48,020 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:55:48,020 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:55:48,020 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:55:48,020 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-20 12:55:48,020 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-20 12:55:48,020 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-20 12:55:48,024 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60420 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-20 12:55:48,024 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): bed4cc4c061ff4314d49ea7add6e3672/B is initiating minor compaction (all files) 2023-04-20 12:55:48,024 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bed4cc4c061ff4314d49ea7add6e3672/B in TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:48,024 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/18ea54d6fc2e4aa3a28a99e7083cd327, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/bdc728ee7c904c85a926570806640adf, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/4d21231fed0a4e63b06b46e442c0cbb2, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/6358f051bdca4ca5baa2d86a1cbc594f, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/662f2d05bf45400289316be46c6d67e2] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp, totalSize=59.0 K 2023-04-20 12:55:48,024 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 69980 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-20 12:55:48,024 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): bed4cc4c061ff4314d49ea7add6e3672/A is initiating minor compaction (all files) 2023-04-20 12:55:48,024 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bed4cc4c061ff4314d49ea7add6e3672/A in TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:48,025 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/be45eb10900f48ac9dafcd62954a767b, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/cc265e856a6c4f4a86619cb3c976e3e5, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/4caa68eee05b480598992e38f1b18389, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/d39746c2ae814a0d83b069e7abcdae2d, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/b66fabb8f0f440a393e596520960776e] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp, totalSize=68.3 K 2023-04-20 12:55:48,025 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 18ea54d6fc2e4aa3a28a99e7083cd327, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1681995344851 2023-04-20 12:55:48,025 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting be45eb10900f48ac9dafcd62954a767b, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1681995344851 2023-04-20 12:55:48,026 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting bdc728ee7c904c85a926570806640adf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1681995345220 2023-04-20 12:55:48,026 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting cc265e856a6c4f4a86619cb3c976e3e5, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1681995345216 2023-04-20 12:55:48,027 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 4d21231fed0a4e63b06b46e442c0cbb2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1681995345382 2023-04-20 12:55:48,027 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 4caa68eee05b480598992e38f1b18389, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1681995345382 2023-04-20 12:55:48,027 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 6358f051bdca4ca5baa2d86a1cbc594f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1681995346019 2023-04-20 12:55:48,027 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting d39746c2ae814a0d83b069e7abcdae2d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1681995346019 2023-04-20 12:55:48,027 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 662f2d05bf45400289316be46c6d67e2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1681995347199 2023-04-20 12:55:48,027 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting b66fabb8f0f440a393e596520960776e, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1681995347199 2023-04-20 12:55:48,040 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): bed4cc4c061ff4314d49ea7add6e3672#B#compaction#415 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:55:48,041 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): bed4cc4c061ff4314d49ea7add6e3672#A#compaction#416 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:55:48,058 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/dd95a4d8a98e4d5ab36b3df2f44f4d34 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/dd95a4d8a98e4d5ab36b3df2f44f4d34 2023-04-20 12:55:48,063 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in bed4cc4c061ff4314d49ea7add6e3672/B of bed4cc4c061ff4314d49ea7add6e3672 into dd95a4d8a98e4d5ab36b3df2f44f4d34(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:55:48,063 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:55:48,063 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672., storeName=bed4cc4c061ff4314d49ea7add6e3672/B, priority=11, startTime=1681995348020; duration=0sec 2023-04-20 12:55:48,063 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:55:48,063 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-20 12:55:48,064 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60420 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-20 12:55:48,064 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): bed4cc4c061ff4314d49ea7add6e3672/C is initiating minor compaction (all files) 2023-04-20 12:55:48,065 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bed4cc4c061ff4314d49ea7add6e3672/C in TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:48,065 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/f45f6f12d5b049b1b6caf833ee52a3ab, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/0ed6702a5dc94f42a8c81f7926d8f219, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/b87fc7f34f124cbc9138305a3b36c026, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/890060bf910542a8ad8f67e0e0c170b6, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/a90d582e84094defbc91492388725ebf] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp, totalSize=59.0 K 2023-04-20 12:55:48,065 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting f45f6f12d5b049b1b6caf833ee52a3ab, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1681995344851 2023-04-20 12:55:48,065 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 0ed6702a5dc94f42a8c81f7926d8f219, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1681995345220 2023-04-20 12:55:48,065 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting b87fc7f34f124cbc9138305a3b36c026, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1681995345382 2023-04-20 12:55:48,066 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 890060bf910542a8ad8f67e0e0c170b6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1681995346019 2023-04-20 12:55:48,066 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting a90d582e84094defbc91492388725ebf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1681995347199 2023-04-20 12:55:48,073 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): bed4cc4c061ff4314d49ea7add6e3672#C#compaction#417 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:55:48,076 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:48,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:48,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:48,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:48,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:48,079 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:48,079 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:48,079 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:48,079 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:48,082 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:48,082 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:48,083 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:48,083 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:48,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:48,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:48,083 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:48,083 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:48,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:48,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:48,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:48,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:48,084 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:48,084 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:48,084 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:48,084 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:48,084 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:48,084 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:48,086 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:48,086 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:48,086 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:48,086 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:48,086 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bed4cc4c061ff4314d49ea7add6e3672 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-20 12:55:48,086 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=A 2023-04-20 12:55:48,086 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:48,086 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=B 2023-04-20 12:55:48,086 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:48,086 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=C 2023-04-20 12:55:48,086 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:48,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on bed4cc4c061ff4314d49ea7add6e3672 2023-04-20 12:55:48,207 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:48,242 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:48,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54244 deadline: 1681995408234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:48,246 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:48,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47054 deadline: 1681995408235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:48,246 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:48,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54262 deadline: 1681995408240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:48,346 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:48,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54244 deadline: 1681995408343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:48,351 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:48,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47054 deadline: 1681995408347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:48,351 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:48,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54262 deadline: 1681995408347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:48,450 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/2e410e3c741342df9498f0977c17702d as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/2e410e3c741342df9498f0977c17702d 2023-04-20 12:55:48,453 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in bed4cc4c061ff4314d49ea7add6e3672/A of bed4cc4c061ff4314d49ea7add6e3672 into 2e410e3c741342df9498f0977c17702d(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:55:48,453 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:55:48,453 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672., storeName=bed4cc4c061ff4314d49ea7add6e3672/A, priority=11, startTime=1681995348020; duration=0sec 2023-04-20 12:55:48,453 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:55:48,486 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/a899124b43be4e738b62bc4dac5b7a19 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/a899124b43be4e738b62bc4dac5b7a19 2023-04-20 12:55:48,489 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in bed4cc4c061ff4314d49ea7add6e3672/C of bed4cc4c061ff4314d49ea7add6e3672 into a899124b43be4e738b62bc4dac5b7a19(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:55:48,489 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:55:48,489 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672., storeName=bed4cc4c061ff4314d49ea7add6e3672/C, priority=11, startTime=1681995348020; duration=0sec 2023-04-20 12:55:48,489 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:55:48,495 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/f541889e4a96431283662bb1fa8d7857 2023-04-20 12:55:48,503 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/b81b0d91142443f0a47cff02f1f11b4b 2023-04-20 12:55:48,550 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:48,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54244 deadline: 1681995408548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:48,554 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:48,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47054 deadline: 1681995408552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:48,554 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:48,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54262 deadline: 1681995408552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:48,855 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:48,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54244 deadline: 1681995408851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:48,858 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:48,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47054 deadline: 1681995408855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:48,858 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:48,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54262 deadline: 1681995408856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:48,910 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/e68164dd50ce481fac6ce5cb4ae53c09 2023-04-20 12:55:48,916 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/f541889e4a96431283662bb1fa8d7857 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/f541889e4a96431283662bb1fa8d7857 2023-04-20 12:55:48,920 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/f541889e4a96431283662bb1fa8d7857, entries=150, sequenceid=230, filesize=11.8 K 2023-04-20 12:55:48,920 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/b81b0d91142443f0a47cff02f1f11b4b as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/b81b0d91142443f0a47cff02f1f11b4b 2023-04-20 12:55:48,932 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/b81b0d91142443f0a47cff02f1f11b4b, entries=150, sequenceid=230, filesize=11.8 K 2023-04-20 12:55:48,933 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/e68164dd50ce481fac6ce5cb4ae53c09 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/e68164dd50ce481fac6ce5cb4ae53c09 2023-04-20 12:55:48,937 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/e68164dd50ce481fac6ce5cb4ae53c09, entries=150, sequenceid=230, filesize=11.8 K 2023-04-20 12:55:48,939 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for bed4cc4c061ff4314d49ea7add6e3672 in 853ms, sequenceid=230, compaction requested=false 2023-04-20 12:55:48,939 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:55:48,939 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:48,939 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:48,939 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:48,939 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:48,942 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:48,942 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:48,943 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:48,943 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:48,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:48,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:48,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:48,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:48,944 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:48,944 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:48,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:48,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:48,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:48,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:48,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:48,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25c25e73[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:48,945 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:48,945 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:48,946 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:48,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:48,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:48,946 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:48,946 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:48,946 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:48,946 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:48,946 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:48,947 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:48,947 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:48,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:48,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:48,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:48,947 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:48,948 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:48,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:48,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:48,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:48,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:48,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:48,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:48,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:48,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:48,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:48,950 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:48,950 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:48,950 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:48,950 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:48,951 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:48,952 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:48,952 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:48,952 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:48,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:48,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:48,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:48,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:48,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:48,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:48,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:48,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:48,952 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:48,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:48,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:48,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:48,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:48,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:48,954 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:48,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:48,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:48,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:48,955 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:48,956 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:48,956 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:48,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:48,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:48,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:48,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:48,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:48,956 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:48,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:48,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:48,956 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:48,956 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:48,956 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:48,956 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:48,956 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:48,956 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:48,956 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:48,956 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:49,056 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:49,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:49,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:49,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:49,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:49,060 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:49,060 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:49,060 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:49,060 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:49,060 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:49,060 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:49,061 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:49,061 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:49,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:49,061 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:49,061 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:49,062 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:49,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:49,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:49,062 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:49,062 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:49,062 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:49,062 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:49,062 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:49,062 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:49,062 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:49,062 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:49,063 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:49,063 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:49,063 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bed4cc4c061ff4314d49ea7add6e3672 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-20 12:55:49,063 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=A 2023-04-20 12:55:49,063 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:49,063 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=B 2023-04-20 12:55:49,063 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:49,063 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=C 2023-04-20 12:55:49,063 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:49,075 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/38e4ad0867124600b853234eb09d7173 2023-04-20 12:55:49,083 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/26aaf31cf51441d79f5faffbe3f1ffcf 2023-04-20 12:55:49,363 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:49,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on bed4cc4c061ff4314d49ea7add6e3672 2023-04-20 12:55:49,406 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:49,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54244 deadline: 1681995409389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,416 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:49,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47054 deadline: 1681995409407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,418 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:49,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54262 deadline: 1681995409408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,493 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/edb995b8299c4ceb94e3d758e4eab0f1 2023-04-20 12:55:49,496 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/38e4ad0867124600b853234eb09d7173 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/38e4ad0867124600b853234eb09d7173 2023-04-20 12:55:49,498 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/38e4ad0867124600b853234eb09d7173, entries=150, sequenceid=250, filesize=11.8 K 2023-04-20 12:55:49,499 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/26aaf31cf51441d79f5faffbe3f1ffcf as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/26aaf31cf51441d79f5faffbe3f1ffcf 2023-04-20 12:55:49,502 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/26aaf31cf51441d79f5faffbe3f1ffcf, entries=150, sequenceid=250, filesize=11.8 K 2023-04-20 12:55:49,502 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/edb995b8299c4ceb94e3d758e4eab0f1 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/edb995b8299c4ceb94e3d758e4eab0f1 2023-04-20 12:55:49,504 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/edb995b8299c4ceb94e3d758e4eab0f1, entries=150, sequenceid=250, filesize=11.8 K 2023-04-20 12:55:49,505 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for bed4cc4c061ff4314d49ea7add6e3672 in 442ms, sequenceid=250, compaction requested=true 2023-04-20 12:55:49,505 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:55:49,505 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:49,505 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:49,505 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:49,505 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,505 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:49,505 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:49,506 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:49,506 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:49,506 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:49,509 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:49,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:49,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:49,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:49,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:49,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:49,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:49,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45ee40f0[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:49,511 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:49,511 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:49,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on bed4cc4c061ff4314d49ea7add6e3672 2023-04-20 12:55:49,511 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:49,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:49,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:49,511 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bed4cc4c061ff4314d49ea7add6e3672 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-20 12:55:49,512 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:49,512 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:49,512 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:49,512 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=A 2023-04-20 12:55:49,512 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:49,512 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=B 2023-04-20 12:55:49,512 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,512 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:49,512 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:49,512 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=C 2023-04-20 12:55:49,512 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:49,512 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:49,512 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,512 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:49,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:49,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:49,513 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:49,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:49,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:49,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:49,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:49,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:49,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:49,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:49,516 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:49,516 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:49,516 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:49,516 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:49,516 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:49,523 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:49,523 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:49,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:49,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:49,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:49,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:49,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:49,524 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,525 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:49,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:49,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:49,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:49,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:49,526 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:49,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:49,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:49,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:49,527 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,528 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:49,528 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:49,528 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:49,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:49,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:49,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:49,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:49,528 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:49,528 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:49,528 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:49,528 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:49,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:49,528 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:49,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:49,528 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:49,528 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:49,528 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:49,546 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:49,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54244 deadline: 1681995409540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,550 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:49,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54262 deadline: 1681995409542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,552 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:49,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47054 deadline: 1681995409547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,628 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:49,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:49,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:49,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:49,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:49,631 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:49,631 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:49,632 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:49,632 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:49,632 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:49,632 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:49,633 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:49,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:49,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:49,633 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:49,633 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,633 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:49,633 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:49,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:49,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:49,634 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:49,634 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:49,634 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:49,634 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:49,634 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:49,634 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:49,634 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:49,634 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:49,634 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:49,634 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:49,634 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:49,634 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:49,634 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:49,634 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:49,634 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,635 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:49,635 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:49,635 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:49,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:49,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:49,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:49,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:49,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,635 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:49,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:49,637 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:49,637 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:49,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:49,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2164993e[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:49,638 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:49,638 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:49,638 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:49,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:49,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:49,638 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:49,638 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:49,638 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,638 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:49,638 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:49,639 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,639 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:49,639 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:49,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:49,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:49,639 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:49,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:49,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:49,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:49,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:49,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:49,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:49,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:49,643 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:49,643 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:49,643 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:49,643 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:49,643 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:49,644 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:49,644 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:49,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:49,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:49,644 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:49,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:49,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:49,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:49,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:49,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:49,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:49,645 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:49,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:49,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:49,650 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:49,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54244 deadline: 1681995409648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:49,650 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:49,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:49,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,657 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,658 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:49,658 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:49,658 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:49,658 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:49,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:49,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:49,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:49,658 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,658 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:49,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:49,658 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:49,658 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:49,658 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:49,658 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:49,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:49,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:49,658 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:49,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:49,659 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:49,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47054 deadline: 1681995409653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,663 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:49,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54262 deadline: 1681995409655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,759 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:49,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:49,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:49,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:49,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:49,761 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:49,761 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:49,761 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:49,761 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:49,761 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:49,761 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:49,762 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:49,762 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:49,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:49,762 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:49,762 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:49,762 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:49,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:49,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:49,763 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:49,763 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:49,763 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:49,763 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:49,763 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:49,763 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:49,763 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:49,763 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:49,763 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:49,763 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:49,763 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:49,763 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:49,763 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:49,763 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:49,763 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,763 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:49,763 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:49,764 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,764 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:49,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:49,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:49,764 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:49,765 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:49,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:49,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:49,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:49,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:49,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:49,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b85ddcb[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:49,766 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:49,766 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:49,766 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,767 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:49,767 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:49,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:49,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:49,767 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:49,767 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:49,767 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:49,767 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,767 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:49,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:49,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:49,768 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:49,768 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:49,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:49,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:49,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:49,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:49,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:49,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:49,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:49,769 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:49,769 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:49,769 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:49,769 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:49,770 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:49,770 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:49,770 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:49,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:49,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:49,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:49,770 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:49,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:49,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:49,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:49,771 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:49,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:49,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:49,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:49,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:49,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:49,772 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:49,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:49,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,773 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:49,773 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:49,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:49,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:49,773 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,773 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:49,773 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:49,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:49,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:49,773 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:49,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:49,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:49,773 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:49,774 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,774 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:49,774 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:49,774 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:49,774 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:49,774 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:49,856 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:49,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54244 deadline: 1681995409852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,869 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:49,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47054 deadline: 1681995409864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,869 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:49,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54262 deadline: 1681995409864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,874 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:49,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:49,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:49,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:49,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:49,877 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:49,877 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:49,877 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:49,877 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:49,877 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:49,877 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:49,878 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,878 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:49,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:49,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:49,878 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,878 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:49,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:49,878 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:49,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:49,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:49,879 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:49,879 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:49,879 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:49,879 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:49,879 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:49,879 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:49,879 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:49,879 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:49,879 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:49,879 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:49,879 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:49,879 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:49,880 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:49,880 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:49,880 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,880 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:49,880 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:49,880 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,880 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:49,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:49,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:49,881 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:49,881 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:49,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:49,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:49,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:49,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:49,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:49,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c83310e[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:49,882 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:49,882 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:49,883 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:49,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:49,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:49,883 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:49,883 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:49,883 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:49,883 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,883 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:49,883 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,883 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:49,883 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:49,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:49,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:49,884 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:49,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:49,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:49,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:49,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:49,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:49,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:49,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:49,885 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:49,885 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:49,885 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:49,885 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:49,885 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:49,888 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:49,889 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:49,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:49,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:49,890 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:49,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:49,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:49,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:49,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:49,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:49,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:49,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:49,891 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:49,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:49,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:49,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,892 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:49,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:49,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,893 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,893 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:49,893 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:49,893 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:49,893 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:49,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:49,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:49,893 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,894 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:49,894 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:49,894 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:49,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:49,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:49,894 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:49,894 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:49,893 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:49,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:49,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:49,926 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=266 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/3c9b511f7299498893955aad9f578ec0 2023-04-20 12:55:49,946 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=266 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/e4b248211dcf414ba2352fa5095aded5 2023-04-20 12:55:49,994 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:49,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:49,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:49,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:49,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:49,998 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:49,998 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:49,998 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:49,998 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:49,998 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:49,998 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:49,999 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,999 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:49,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:49,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:49,999 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:49,999 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:49,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:50,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:50,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:50,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:50,000 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:50,000 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:50,000 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:50,000 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:50,000 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:50,000 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:50,006 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:50,006 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:50,006 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:50,006 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:50,006 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:50,006 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:50,006 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:50,006 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:50,006 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,006 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:50,006 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:50,007 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:50,007 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:50,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:50,007 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:50,008 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:50,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:50,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:50,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:50,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:50,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:50,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bdd05ae[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:50,008 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:50,009 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:50,009 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:50,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:50,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:50,009 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,009 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:50,009 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:50,009 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:50,009 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:50,010 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,010 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:50,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:50,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:50,010 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:50,010 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:50,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:50,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:50,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:50,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:50,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:50,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:50,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:50,012 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:50,012 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:50,012 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:50,012 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:50,012 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:50,013 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:50,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:50,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:50,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:50,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:50,013 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:50,014 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:50,014 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:50,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:50,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:50,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:50,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:50,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:50,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:50,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,014 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:50,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:50,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:50,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,016 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,016 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:50,016 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:50,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:50,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:50,017 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:50,017 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:50,017 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,017 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:50,017 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:50,017 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:50,017 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:50,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:50,016 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:50,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:50,017 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:50,017 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:50,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:50,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:50,117 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:50,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:50,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:50,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:50,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:50,120 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:50,120 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:50,120 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:50,120 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:50,120 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:50,120 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:50,121 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:50,121 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:50,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:50,121 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,121 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:50,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:50,121 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:50,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:50,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:50,122 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:50,122 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:50,122 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:50,122 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:50,122 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:50,122 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:50,122 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:50,122 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:50,122 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:50,122 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:50,122 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:50,122 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:50,122 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:50,122 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:50,122 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,122 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:50,123 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:50,123 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:50,123 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:50,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:50,124 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:50,124 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:50,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:50,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:50,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:50,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:50,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:50,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ef5ca82[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:50,125 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:50,125 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:50,126 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:50,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:50,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:50,126 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:50,126 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:50,126 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,126 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:50,126 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:50,127 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:50,127 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:50,127 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,127 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:50,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:50,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:50,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:50,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:50,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:50,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:50,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:50,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:50,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:50,130 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:50,130 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:50,130 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:50,130 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:50,130 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:50,131 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:50,131 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:50,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:50,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:50,132 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:50,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:50,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:50,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:50,132 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,133 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:50,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:50,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:50,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:50,135 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:50,135 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:50,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:50,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:50,135 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,135 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:50,135 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:50,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:50,135 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:50,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:50,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:50,135 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,135 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:50,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:50,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:50,135 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:50,135 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:50,135 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:50,135 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:50,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:50,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:50,161 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:50,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54244 deadline: 1681995410158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,174 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:50,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47054 deadline: 1681995410172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,175 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:50,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54262 deadline: 1681995410172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,236 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:50,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:50,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:50,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:50,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:50,239 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:50,239 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:50,239 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:50,239 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:50,239 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:50,239 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:50,240 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:50,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:50,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:50,240 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:50,240 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,240 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:50,240 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:50,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:50,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:50,240 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:50,241 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:50,241 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:50,241 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:50,241 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:50,241 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:50,242 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:50,242 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:50,242 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:50,242 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:50,242 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:50,242 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:50,242 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:50,242 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:50,242 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,242 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:50,242 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:50,243 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,243 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:50,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:50,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:50,243 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:50,243 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:50,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:50,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:50,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:50,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:50,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:50,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b6b3f12[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:50,245 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:50,245 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:50,245 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:50,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:50,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:50,245 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:50,245 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,245 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:50,245 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:50,245 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:50,246 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,246 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:50,246 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:50,246 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:50,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:50,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:50,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:50,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:50,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:50,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:50,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:50,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:50,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:50,250 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:50,250 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:50,250 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:50,250 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:50,250 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:50,256 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:50,256 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:50,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:50,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:50,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:50,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:50,257 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:50,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:50,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:50,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:50,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:50,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:50,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:50,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:50,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:50,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:50,258 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:50,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,259 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,261 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,261 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:50,261 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:50,261 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:50,261 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:50,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:50,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:50,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:50,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:50,261 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,261 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:50,261 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:50,261 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:50,261 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:50,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:50,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:50,261 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:50,261 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:50,360 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=266 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/6ea823ef6a854ad4afb010a2e7ae12a9 2023-04-20 12:55:50,362 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:50,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:50,364 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/3c9b511f7299498893955aad9f578ec0 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/3c9b511f7299498893955aad9f578ec0 2023-04-20 12:55:50,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:50,367 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/3c9b511f7299498893955aad9f578ec0, entries=200, sequenceid=266, filesize=14.3 K 2023-04-20 12:55:50,368 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/e4b248211dcf414ba2352fa5095aded5 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/e4b248211dcf414ba2352fa5095aded5 2023-04-20 12:55:50,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:50,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:50,370 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:50,370 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:50,370 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:50,370 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:50,370 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:50,370 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:50,371 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:50,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:50,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:50,371 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:50,372 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,372 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:50,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:50,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:50,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:50,372 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:50,372 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/e4b248211dcf414ba2352fa5095aded5, entries=150, sequenceid=266, filesize=11.9 K 2023-04-20 12:55:50,373 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/6ea823ef6a854ad4afb010a2e7ae12a9 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/6ea823ef6a854ad4afb010a2e7ae12a9 2023-04-20 12:55:50,374 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:50,374 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:50,374 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:50,374 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:50,374 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:50,379 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:50,379 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:50,379 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:50,380 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:50,380 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:50,381 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:50,381 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:50,381 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:50,381 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,381 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:50,381 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:50,382 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/6ea823ef6a854ad4afb010a2e7ae12a9, entries=150, sequenceid=266, filesize=11.9 K 2023-04-20 12:55:50,383 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for bed4cc4c061ff4314d49ea7add6e3672 in 872ms, sequenceid=266, compaction requested=true 2023-04-20 12:55:50,383 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:55:50,383 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:55:50,383 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:55:50,383 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:55:50,383 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 12:55:50,383 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 12:55:50,384 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51144 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 12:55:50,384 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48704 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 12:55:50,384 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): bed4cc4c061ff4314d49ea7add6e3672/A is initiating minor compaction (all files) 2023-04-20 12:55:50,384 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): bed4cc4c061ff4314d49ea7add6e3672/B is initiating minor compaction (all files) 2023-04-20 12:55:50,384 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bed4cc4c061ff4314d49ea7add6e3672/A in TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:50,385 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bed4cc4c061ff4314d49ea7add6e3672/B in TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:50,385 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/dd95a4d8a98e4d5ab36b3df2f44f4d34, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/b81b0d91142443f0a47cff02f1f11b4b, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/26aaf31cf51441d79f5faffbe3f1ffcf, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/e4b248211dcf414ba2352fa5095aded5] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp, totalSize=47.6 K 2023-04-20 12:55:50,385 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/2e410e3c741342df9498f0977c17702d, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/f541889e4a96431283662bb1fa8d7857, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/38e4ad0867124600b853234eb09d7173, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/3c9b511f7299498893955aad9f578ec0] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp, totalSize=49.9 K 2023-04-20 12:55:50,385 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting dd95a4d8a98e4d5ab36b3df2f44f4d34, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1681995347199 2023-04-20 12:55:50,385 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 2e410e3c741342df9498f0977c17702d, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1681995347199 2023-04-20 12:55:50,385 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting b81b0d91142443f0a47cff02f1f11b4b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1681995347558 2023-04-20 12:55:50,385 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting f541889e4a96431283662bb1fa8d7857, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1681995347558 2023-04-20 12:55:50,385 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 26aaf31cf51441d79f5faffbe3f1ffcf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1681995348231 2023-04-20 12:55:50,385 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 38e4ad0867124600b853234eb09d7173, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1681995348231 2023-04-20 12:55:50,386 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting e4b248211dcf414ba2352fa5095aded5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=266, earliestPutTs=1681995349375 2023-04-20 12:55:50,386 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 3c9b511f7299498893955aad9f578ec0, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=266, earliestPutTs=1681995349375 2023-04-20 12:55:50,386 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:50,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:50,386 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:50,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:50,386 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:50,386 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:50,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:50,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:50,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:50,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:50,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e957030[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:50,388 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:50,388 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:50,390 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:50,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:50,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:50,390 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:50,390 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:50,390 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:50,390 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,390 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:50,391 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,391 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:50,391 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:50,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:50,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:50,391 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:50,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:50,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:50,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:50,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:50,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:50,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:50,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:50,394 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:50,394 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:50,394 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:50,394 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:50,394 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:50,396 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): bed4cc4c061ff4314d49ea7add6e3672#A#compaction#427 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:55:50,399 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:50,399 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:50,399 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:50,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:50,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:50,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:50,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:50,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:50,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:50,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:50,402 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:50,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:50,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:50,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:50,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:50,404 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:50,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:50,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,411 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:50,411 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:50,411 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:50,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:50,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:50,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:50,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:50,411 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:50,411 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:50,411 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:50,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:50,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:50,412 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): bed4cc4c061ff4314d49ea7add6e3672#B#compaction#428 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:55:50,412 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,412 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:50,412 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:50,413 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,413 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:50,413 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:50,413 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:50,447 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/3d060eea3def4f318ee361534c53e147 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/3d060eea3def4f318ee361534c53e147 2023-04-20 12:55:50,450 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bed4cc4c061ff4314d49ea7add6e3672/B of bed4cc4c061ff4314d49ea7add6e3672 into 3d060eea3def4f318ee361534c53e147(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:55:50,450 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:55:50,450 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672., storeName=bed4cc4c061ff4314d49ea7add6e3672/B, priority=12, startTime=1681995350383; duration=0sec 2023-04-20 12:55:50,450 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:55:50,451 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 12:55:50,451 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48704 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 12:55:50,451 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): bed4cc4c061ff4314d49ea7add6e3672/C is initiating minor compaction (all files) 2023-04-20 12:55:50,452 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bed4cc4c061ff4314d49ea7add6e3672/C in TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:50,452 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/a899124b43be4e738b62bc4dac5b7a19, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/e68164dd50ce481fac6ce5cb4ae53c09, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/edb995b8299c4ceb94e3d758e4eab0f1, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/6ea823ef6a854ad4afb010a2e7ae12a9] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp, totalSize=47.6 K 2023-04-20 12:55:50,452 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting a899124b43be4e738b62bc4dac5b7a19, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=215, earliestPutTs=1681995347199 2023-04-20 12:55:50,452 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting e68164dd50ce481fac6ce5cb4ae53c09, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1681995347558 2023-04-20 12:55:50,452 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting edb995b8299c4ceb94e3d758e4eab0f1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1681995348231 2023-04-20 12:55:50,453 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 6ea823ef6a854ad4afb010a2e7ae12a9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=266, earliestPutTs=1681995349375 2023-04-20 12:55:50,457 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): bed4cc4c061ff4314d49ea7add6e3672#C#compaction#429 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:55:50,470 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/ba43d43ba99d4f16831f97be463ccee0 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/ba43d43ba99d4f16831f97be463ccee0 2023-04-20 12:55:50,474 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bed4cc4c061ff4314d49ea7add6e3672/C of bed4cc4c061ff4314d49ea7add6e3672 into ba43d43ba99d4f16831f97be463ccee0(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:55:50,474 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:55:50,474 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672., storeName=bed4cc4c061ff4314d49ea7add6e3672/C, priority=12, startTime=1681995350383; duration=0sec 2023-04-20 12:55:50,474 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:55:50,511 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:50,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:50,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:50,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:50,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:50,515 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:50,515 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:50,515 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:50,515 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:50,515 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:50,515 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:50,516 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,516 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:50,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:50,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:50,516 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,516 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:50,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:50,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:50,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:50,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:50,517 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:50,517 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:50,517 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:50,517 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:50,517 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:50,517 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:50,520 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:50,520 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:50,520 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:50,520 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:50,520 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bed4cc4c061ff4314d49ea7add6e3672 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-20 12:55:50,521 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=A 2023-04-20 12:55:50,521 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:50,521 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=B 2023-04-20 12:55:50,521 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:50,521 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=C 2023-04-20 12:55:50,521 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:50,528 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=284 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/b8b462643aaa488fa22686eb00de3e72 2023-04-20 12:55:50,543 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=284 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/7bf9bf4cd0cd4983a354c2c1021f2ef6 2023-04-20 12:55:50,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on bed4cc4c061ff4314d49ea7add6e3672 2023-04-20 12:55:50,670 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:50,697 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:50,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54262 deadline: 1681995410695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,703 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:50,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47054 deadline: 1681995410696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,703 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:50,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54244 deadline: 1681995410697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,802 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:50,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54262 deadline: 1681995410798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:50,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54244 deadline: 1681995410804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:50,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47054 deadline: 1681995410804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,829 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/36514fed58694c39a437fe4699331219 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/36514fed58694c39a437fe4699331219 2023-04-20 12:55:50,834 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bed4cc4c061ff4314d49ea7add6e3672/A of bed4cc4c061ff4314d49ea7add6e3672 into 36514fed58694c39a437fe4699331219(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:55:50,834 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:55:50,834 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672., storeName=bed4cc4c061ff4314d49ea7add6e3672/A, priority=12, startTime=1681995350383; duration=0sec 2023-04-20 12:55:50,835 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:55:50,951 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=284 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/4c486af44e7645578cb9a672a5cd6880 2023-04-20 12:55:50,955 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/b8b462643aaa488fa22686eb00de3e72 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/b8b462643aaa488fa22686eb00de3e72 2023-04-20 12:55:50,958 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/b8b462643aaa488fa22686eb00de3e72, entries=150, sequenceid=284, filesize=11.9 K 2023-04-20 12:55:50,959 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/7bf9bf4cd0cd4983a354c2c1021f2ef6 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/7bf9bf4cd0cd4983a354c2c1021f2ef6 2023-04-20 12:55:50,971 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/7bf9bf4cd0cd4983a354c2c1021f2ef6, entries=150, sequenceid=284, filesize=11.9 K 2023-04-20 12:55:50,972 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/4c486af44e7645578cb9a672a5cd6880 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/4c486af44e7645578cb9a672a5cd6880 2023-04-20 12:55:50,975 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/4c486af44e7645578cb9a672a5cd6880, entries=150, sequenceid=284, filesize=11.9 K 2023-04-20 12:55:50,976 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for bed4cc4c061ff4314d49ea7add6e3672 in 456ms, sequenceid=284, compaction requested=false 2023-04-20 12:55:50,976 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:55:50,976 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:50,976 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:50,976 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:50,976 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,976 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:50,976 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:50,978 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:50,978 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:50,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:50,978 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:50,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:50,978 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:50,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:50,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:50,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:50,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:50,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6dd6ee39[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:50,979 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:50,979 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:50,980 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:50,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:50,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:50,980 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:50,980 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:50,980 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,980 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:50,980 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:50,981 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,981 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:50,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:50,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:50,981 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:50,981 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:50,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:50,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:50,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:50,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:50,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:50,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:50,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:50,982 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:50,982 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:50,982 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:50,982 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:50,982 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:50,983 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:50,983 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:50,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:50,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:50,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:50,983 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:50,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:50,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:50,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:50,984 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:50,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:50,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:50,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:50,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:50,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:50,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,986 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:50,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:50,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,987 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,987 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:50,987 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:50,987 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:50,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:50,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:50,987 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:50,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:50,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:50,987 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:50,987 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:50,987 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:50,987 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:50,987 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:50,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:50,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:50,987 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:50,987 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on bed4cc4c061ff4314d49ea7add6e3672 2023-04-20 12:55:51,008 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bed4cc4c061ff4314d49ea7add6e3672 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-20 12:55:51,009 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=A 2023-04-20 12:55:51,009 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:51,009 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=B 2023-04-20 12:55:51,009 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:51,009 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=C 2023-04-20 12:55:51,009 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:51,041 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:51,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54262 deadline: 1681995411034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,044 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:51,044 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:51,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54244 deadline: 1681995411038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47054 deadline: 1681995411039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,087 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:51,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:51,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:51,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:51,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:51,090 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:51,090 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:51,090 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:51,090 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:51,090 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,090 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:51,091 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:51,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:51,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:51,091 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:51,091 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,091 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:51,091 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:51,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:51,091 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:51,092 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:51,092 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:51,092 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:51,092 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:51,092 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:51,092 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:51,092 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:51,092 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:51,092 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:51,092 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:51,092 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:51,092 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:51,092 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:51,092 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,092 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:51,092 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:51,094 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,094 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:51,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:51,094 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,094 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:51,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:51,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:51,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:51,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:51,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:51,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d0798db[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:51,097 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:51,097 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,098 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,098 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:51,098 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:51,098 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:51,098 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,098 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:51,099 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:51,099 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:51,099 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:51,099 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:51,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:51,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:51,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:51,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:51,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:51,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:51,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:51,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:51,101 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:51,101 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:51,101 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:51,101 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:51,101 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:51,103 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,103 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,103 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:51,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:51,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:51,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:51,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:51,104 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:51,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:51,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:51,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:51,105 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:51,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:51,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,106 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,106 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:51,106 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:51,107 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:51,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:51,107 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:51,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:51,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:51,107 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:51,107 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:51,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:51,107 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,107 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,107 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:51,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:51,107 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:51,107 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:51,148 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:51,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54262 deadline: 1681995411143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,150 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:51,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47054 deadline: 1681995411146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,150 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:51,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54244 deadline: 1681995411146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,207 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:51,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:51,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:51,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:51,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:51,211 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:51,211 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:51,211 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:51,211 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:51,211 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,211 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:51,212 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:51,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:51,212 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:51,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:51,213 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:51,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:51,213 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:51,213 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,213 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:51,214 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:51,214 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:51,214 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:51,214 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:51,214 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:51,215 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:51,216 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:51,216 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:51,216 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:51,216 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:51,216 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:51,216 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:51,216 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:51,216 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,216 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:51,216 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:51,223 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:51,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:51,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:51,224 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:51,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:51,224 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,224 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:51,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:51,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:51,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fe6a7c6[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:51,225 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:51,225 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,226 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,226 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:51,226 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:51,226 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:51,227 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:51,227 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:51,227 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:51,227 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,227 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:51,228 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:51,228 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:51,228 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:51,228 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:51,228 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:51,228 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:51,229 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,229 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:51,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:51,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,229 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:51,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:51,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:51,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:51,230 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:51,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:51,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:51,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,233 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:51,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:51,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,237 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,237 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:51,237 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:51,237 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,237 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,237 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,237 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,237 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:51,237 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:51,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:51,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:51,237 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:51,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:51,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:51,237 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:51,237 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:51,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:51,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:51,276 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:51,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:54242 deadline: 1681995411270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,278 DEBUG [Thread-3079] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8199 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672., hostname=jenkins-hbase20.apache.org,36277,1681995213947, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-20 12:55:51,284 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:51,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:47068 deadline: 1681995411283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,287 DEBUG [Thread-3087] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8202 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672., hostname=jenkins-hbase20.apache.org,36277,1681995213947, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-20 12:55:51,338 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:51,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:51,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:51,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:51,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:51,342 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:51,342 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:51,342 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:51,342 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:51,343 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,343 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:51,346 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:51,346 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:51,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:51,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:51,347 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,347 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,347 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:51,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:51,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:51,347 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:51,350 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:51,350 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:51,350 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:51,350 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:51,350 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:51,354 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:51,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54262 deadline: 1681995411349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,354 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:51,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47054 deadline: 1681995411351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,355 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:51,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54244 deadline: 1681995411352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,357 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:51,357 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:51,359 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:51,359 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:51,359 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:51,359 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:51,360 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:51,360 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:51,360 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,360 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:51,360 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:51,361 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,361 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:51,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:51,361 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,361 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:51,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:51,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:51,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:51,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:51,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:51,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a874efa[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:51,362 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:51,363 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,365 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,365 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,365 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:51,365 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:51,365 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:51,365 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:51,366 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,366 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:51,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:51,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:51,366 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:51,367 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:51,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:51,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:51,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:51,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:51,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:51,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:51,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:51,368 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:51,368 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:51,368 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:51,368 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:51,368 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:51,369 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,369 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,369 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:51,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:51,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:51,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:51,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:51,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:51,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,370 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:51,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:51,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:51,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:51,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:51,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,372 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,374 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:51,374 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:51,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:51,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:51,374 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:51,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:51,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:51,374 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,374 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:51,374 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:51,374 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,374 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,374 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,374 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,374 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:51,374 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:51,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:51,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:51,423 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=302 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/633cc9c9743c4d2fb064b7f3e4eaf52b 2023-04-20 12:55:51,474 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:51,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:51,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:51,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:51,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:51,478 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:51,478 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:51,478 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:51,478 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:51,479 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,479 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:51,479 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:51,479 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:51,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:51,480 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:51,480 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:51,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:51,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:51,480 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:51,489 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:51,489 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:51,489 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:51,489 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:51,489 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:51,494 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:51,494 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:51,494 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:51,494 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:51,494 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:51,494 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:51,494 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:51,494 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:51,494 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,494 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:51,494 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:51,496 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,496 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:51,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:51,496 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,496 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:51,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:51,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:51,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:51,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:51,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:51,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40faccb5[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:51,497 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:51,497 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,498 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,498 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:51,498 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:51,498 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:51,498 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,498 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:51,499 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:51,499 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:51,499 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:51,499 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:51,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:51,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:51,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:51,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:51,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:51,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:51,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:51,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:51,502 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:51,502 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:51,502 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:51,502 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:51,502 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:51,503 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,503 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,503 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:51,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:51,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:51,504 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:51,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:51,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:51,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:51,505 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:51,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:51,507 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:51,507 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:51,507 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:51,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:51,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:51,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:51,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:51,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:51,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:51,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:51,508 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:51,508 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:51,508 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,508 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:51,508 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:51,508 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:51,508 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,508 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,508 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,508 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,608 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:51,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:51,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:51,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:51,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:51,611 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:51,611 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:51,611 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:51,611 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:51,611 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,611 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:51,612 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:51,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:51,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:51,612 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:51,612 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,612 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:51,612 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:51,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:51,613 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:51,613 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:51,613 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:51,613 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:51,613 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:51,613 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:51,618 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:51,618 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:51,618 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:51,618 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:51,618 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:51,618 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:51,618 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:51,618 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:51,618 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,618 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:51,618 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:51,623 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,623 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:51,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:51,623 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,623 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:51,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:51,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:51,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:51,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:51,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:51,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@611a5531[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:51,624 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:51,624 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,625 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,625 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:51,625 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:51,625 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:51,625 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,625 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:51,626 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,626 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:51,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:51,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:51,626 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:51,626 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:51,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:51,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:51,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:51,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:51,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:51,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:51,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:51,627 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:51,627 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:51,627 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:51,627 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:51,627 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:51,628 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,628 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,628 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:51,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:51,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:51,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:51,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:51,629 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:51,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:51,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:51,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:51,630 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:51,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:51,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,632 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,632 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:51,632 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:51,632 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,632 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,632 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:51,632 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,632 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:51,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:51,632 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:51,632 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:51,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:51,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:51,632 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:51,632 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:51,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:51,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:51,657 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:51,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54244 deadline: 1681995411656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,657 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:51,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47054 deadline: 1681995411656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,665 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:51,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54262 deadline: 1681995411657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,733 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:51,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:51,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:51,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:51,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:51,735 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:51,735 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:51,735 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:51,735 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:51,735 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,735 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:51,736 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:51,736 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:51,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:51,736 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,736 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:51,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:51,737 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:51,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:51,737 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:51,737 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:51,737 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:51,737 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:51,737 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:51,737 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:51,737 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:51,737 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:51,737 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:51,737 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:51,738 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:51,738 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:51,738 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:51,738 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:51,738 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,738 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:51,738 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:51,738 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,739 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:51,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:51,739 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,739 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:51,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:51,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:51,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:51,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:51,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:51,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b980643[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:51,740 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:51,740 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,740 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,740 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:51,740 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:51,740 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,741 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:51,741 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:51,743 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,743 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:51,743 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:51,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:51,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:51,743 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:51,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:51,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:51,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:51,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:51,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:51,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:51,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:51,744 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:51,744 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:51,744 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:51,745 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:51,745 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:51,745 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,745 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,745 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:51,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:51,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:51,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:51,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:51,747 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:51,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:51,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:51,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:51,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,748 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:51,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:51,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,750 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,750 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:51,750 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:51,750 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:51,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:51,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:51,750 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:51,750 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:51,750 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:51,750 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:51,750 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:51,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:51,750 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,750 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,750 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:51,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:51,845 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=302 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/5bf62e40ef604331ad398d58ea0c954a 2023-04-20 12:55:51,850 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:51,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:51,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:51,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:51,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:51,853 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:51,853 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:51,853 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=302 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/561cb9b4e85b4c819ef6e65b206adb5e 2023-04-20 12:55:51,853 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:51,853 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:51,854 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,854 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:51,854 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:51,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:51,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:51,854 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:51,855 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,855 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:51,855 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:51,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:51,855 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:51,855 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:51,855 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:51,856 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:51,856 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:51,856 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:51,856 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:51,856 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:51,856 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:51,856 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:51,856 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:51,856 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:51,856 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:51,856 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:51,856 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/633cc9c9743c4d2fb064b7f3e4eaf52b as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/633cc9c9743c4d2fb064b7f3e4eaf52b 2023-04-20 12:55:51,856 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,856 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:51,856 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:51,857 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,857 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:51,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:51,857 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,858 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:51,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:51,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:51,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:51,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:51,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:51,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b487f54[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:51,859 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:51,859 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,860 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/633cc9c9743c4d2fb064b7f3e4eaf52b, entries=200, sequenceid=302, filesize=14.3 K 2023-04-20 12:55:51,860 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,860 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:51,860 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,860 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:51,860 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:51,860 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:51,860 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/5bf62e40ef604331ad398d58ea0c954a as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/5bf62e40ef604331ad398d58ea0c954a 2023-04-20 12:55:51,861 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:51,861 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:51,861 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:51,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:51,861 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:51,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:51,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:51,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:51,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:51,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:51,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:51,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:51,863 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:51,863 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:51,863 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:51,863 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/5bf62e40ef604331ad398d58ea0c954a, entries=150, sequenceid=302, filesize=11.9 K 2023-04-20 12:55:51,863 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:51,863 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:51,863 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/561cb9b4e85b4c819ef6e65b206adb5e as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/561cb9b4e85b4c819ef6e65b206adb5e 2023-04-20 12:55:51,864 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,864 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:51,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:51,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,864 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:51,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:51,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:51,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:51,866 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:51,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:51,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:51,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,867 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:51,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:51,867 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/561cb9b4e85b4c819ef6e65b206adb5e, entries=150, sequenceid=302, filesize=11.9 K 2023-04-20 12:55:51,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,868 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for bed4cc4c061ff4314d49ea7add6e3672 in 860ms, sequenceid=302, compaction requested=true 2023-04-20 12:55:51,868 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:55:51,868 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,868 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:51,868 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:51,868 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:55:51,868 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:51,868 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:55:51,868 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:51,868 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:55:51,868 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:51,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:51,868 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:55:51,868 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:51,869 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,869 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:55:51,869 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:51,869 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:51,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:51,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:51,869 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:51,869 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:51,869 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:51,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:51,869 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:55:51,869 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39544 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:55:51,870 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): bed4cc4c061ff4314d49ea7add6e3672/B is initiating minor compaction (all files) 2023-04-20 12:55:51,870 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bed4cc4c061ff4314d49ea7add6e3672/B in TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:51,870 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): bed4cc4c061ff4314d49ea7add6e3672/A is initiating minor compaction (all files) 2023-04-20 12:55:51,870 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bed4cc4c061ff4314d49ea7add6e3672/A in TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:51,870 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/36514fed58694c39a437fe4699331219, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/b8b462643aaa488fa22686eb00de3e72, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/633cc9c9743c4d2fb064b7f3e4eaf52b] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp, totalSize=38.6 K 2023-04-20 12:55:51,870 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/3d060eea3def4f318ee361534c53e147, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/7bf9bf4cd0cd4983a354c2c1021f2ef6, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/5bf62e40ef604331ad398d58ea0c954a] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp, totalSize=36.2 K 2023-04-20 12:55:51,870 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 36514fed58694c39a437fe4699331219, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=266, earliestPutTs=1681995349375 2023-04-20 12:55:51,870 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 3d060eea3def4f318ee361534c53e147, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=266, earliestPutTs=1681995349375 2023-04-20 12:55:51,870 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting b8b462643aaa488fa22686eb00de3e72, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1681995349539 2023-04-20 12:55:51,872 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 7bf9bf4cd0cd4983a354c2c1021f2ef6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1681995349539 2023-04-20 12:55:51,872 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 633cc9c9743c4d2fb064b7f3e4eaf52b, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1681995350688 2023-04-20 12:55:51,872 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 5bf62e40ef604331ad398d58ea0c954a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1681995350688 2023-04-20 12:55:51,876 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): bed4cc4c061ff4314d49ea7add6e3672#A#compaction#436 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:55:51,877 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): bed4cc4c061ff4314d49ea7add6e3672#B#compaction#437 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:55:51,969 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:51,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:51,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:51,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:51,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:51,972 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:51,972 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:51,972 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:51,972 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:51,972 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,972 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:51,973 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,973 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:51,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:51,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:51,973 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:51,973 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:51,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:51,973 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:51,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:51,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:51,974 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:51,974 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:51,974 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:51,974 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:51,974 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:51,974 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:51,974 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:51,974 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:51,974 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:51,974 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:51,974 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bed4cc4c061ff4314d49ea7add6e3672 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-20 12:55:51,975 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=A 2023-04-20 12:55:51,975 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:51,975 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=B 2023-04-20 12:55:51,975 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:51,975 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=C 2023-04-20 12:55:51,975 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:52,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on bed4cc4c061ff4314d49ea7add6e3672 2023-04-20 12:55:52,160 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:52,191 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:52,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47054 deadline: 1681995412187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:52,193 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:52,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54244 deadline: 1681995412188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:52,193 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:52,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54262 deadline: 1681995412188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:52,288 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/1f7bd7186f4f453e9a203b651a3e4dc8 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/1f7bd7186f4f453e9a203b651a3e4dc8 2023-04-20 12:55:52,289 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/c952d98f537e4d9aa11dd71394d87f50 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/c952d98f537e4d9aa11dd71394d87f50 2023-04-20 12:55:52,293 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:52,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47054 deadline: 1681995412292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:52,294 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:52,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54244 deadline: 1681995412294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:52,295 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:52,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54262 deadline: 1681995412294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:52,302 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bed4cc4c061ff4314d49ea7add6e3672/A of bed4cc4c061ff4314d49ea7add6e3672 into c952d98f537e4d9aa11dd71394d87f50(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:55:52,302 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:55:52,302 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672., storeName=bed4cc4c061ff4314d49ea7add6e3672/A, priority=13, startTime=1681995351868; duration=0sec 2023-04-20 12:55:52,302 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:55:52,302 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:55:52,303 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:55:52,303 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): bed4cc4c061ff4314d49ea7add6e3672/C is initiating minor compaction (all files) 2023-04-20 12:55:52,304 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bed4cc4c061ff4314d49ea7add6e3672/C in TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:52,304 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/ba43d43ba99d4f16831f97be463ccee0, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/4c486af44e7645578cb9a672a5cd6880, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/561cb9b4e85b4c819ef6e65b206adb5e] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp, totalSize=36.2 K 2023-04-20 12:55:52,304 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting ba43d43ba99d4f16831f97be463ccee0, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=266, earliestPutTs=1681995349375 2023-04-20 12:55:52,304 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 4c486af44e7645578cb9a672a5cd6880, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1681995349539 2023-04-20 12:55:52,305 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 561cb9b4e85b4c819ef6e65b206adb5e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1681995350688 2023-04-20 12:55:52,319 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): bed4cc4c061ff4314d49ea7add6e3672#C#compaction#439 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:55:52,319 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bed4cc4c061ff4314d49ea7add6e3672/B of bed4cc4c061ff4314d49ea7add6e3672 into 1f7bd7186f4f453e9a203b651a3e4dc8(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:55:52,319 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:55:52,319 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672., storeName=bed4cc4c061ff4314d49ea7add6e3672/B, priority=13, startTime=1681995351868; duration=0sec 2023-04-20 12:55:52,320 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:55:52,351 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/8b9a4a865144454b964ffc775dd2889e as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/8b9a4a865144454b964ffc775dd2889e 2023-04-20 12:55:52,359 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bed4cc4c061ff4314d49ea7add6e3672/C of bed4cc4c061ff4314d49ea7add6e3672 into 8b9a4a865144454b964ffc775dd2889e(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:55:52,359 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:55:52,359 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672., storeName=bed4cc4c061ff4314d49ea7add6e3672/C, priority=13, startTime=1681995351869; duration=0sec 2023-04-20 12:55:52,359 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:55:52,384 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/e6a756def45a438c8751f0461bb2612d 2023-04-20 12:55:52,496 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:52,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47054 deadline: 1681995412495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:52,500 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:52,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54244 deadline: 1681995412497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:52,501 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:52,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54262 deadline: 1681995412498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:52,796 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/5d2c401500a141ee83820384ad2be706 2023-04-20 12:55:52,802 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:52,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47054 deadline: 1681995412799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:52,805 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:52,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54244 deadline: 1681995412803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:52,807 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:52,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54262 deadline: 1681995412804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,206 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/12e870813b964fab963bd7b327c4361d 2023-04-20 12:55:53,210 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/e6a756def45a438c8751f0461bb2612d as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/e6a756def45a438c8751f0461bb2612d 2023-04-20 12:55:53,216 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/e6a756def45a438c8751f0461bb2612d, entries=150, sequenceid=317, filesize=11.9 K 2023-04-20 12:55:53,216 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/5d2c401500a141ee83820384ad2be706 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/5d2c401500a141ee83820384ad2be706 2023-04-20 12:55:53,219 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/5d2c401500a141ee83820384ad2be706, entries=150, sequenceid=317, filesize=11.9 K 2023-04-20 12:55:53,220 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/12e870813b964fab963bd7b327c4361d as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/12e870813b964fab963bd7b327c4361d 2023-04-20 12:55:53,227 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/12e870813b964fab963bd7b327c4361d, entries=150, sequenceid=317, filesize=11.9 K 2023-04-20 12:55:53,228 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for bed4cc4c061ff4314d49ea7add6e3672 in 1254ms, sequenceid=317, compaction requested=false 2023-04-20 12:55:53,228 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:55:53,228 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:53,228 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:53,228 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:53,229 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,229 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:53,229 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:53,230 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,230 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:53,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:53,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:53,230 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:53,231 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:53,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:53,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:53,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:53,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:53,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:53,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dc197ec[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:53,232 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:53,232 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:53,233 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:53,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:53,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:53,233 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:53,233 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:53,233 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:53,233 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,233 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:53,234 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,234 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:53,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:53,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:53,234 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:53,234 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:53,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:53,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:53,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:53,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:53,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:53,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:53,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:53,237 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:53,237 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:53,238 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:53,238 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:53,238 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:53,240 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:53,240 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:53,240 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:53,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:53,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:53,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:53,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:53,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:53,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:53,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:53,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:53,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:53,241 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:53,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:53,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:53,243 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:53,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:53,248 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,248 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:53,248 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:53,248 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:53,248 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:53,248 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:53,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:53,248 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:53,249 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:53,249 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:53,248 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:53,249 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:53,249 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:53,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:53,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:53,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:53,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:53,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:53,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on bed4cc4c061ff4314d49ea7add6e3672 2023-04-20 12:55:53,311 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bed4cc4c061ff4314d49ea7add6e3672 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-20 12:55:53,312 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=A 2023-04-20 12:55:53,312 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:53,312 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=B 2023-04-20 12:55:53,312 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:53,312 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=C 2023-04-20 12:55:53,312 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:53,320 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=340 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/9b4b7ac06f9346bea6e3a44142aefb6b 2023-04-20 12:55:53,332 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=340 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/8575608848b341f398ce7b5adea8980a 2023-04-20 12:55:53,337 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:53,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54262 deadline: 1681995413332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,337 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:53,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47054 deadline: 1681995413332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,338 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:53,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54244 deadline: 1681995413333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,349 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:53,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:53,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:53,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:53,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:53,351 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:53,351 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:53,351 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:53,351 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:53,352 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:53,352 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:53,352 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,352 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:53,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:53,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:53,353 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,353 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:53,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:53,353 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:53,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:53,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:53,353 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:53,354 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:53,354 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:53,354 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:53,354 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:53,354 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:53,354 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:53,354 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:53,354 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:53,354 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:53,354 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:53,354 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:53,354 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:53,354 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,354 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:53,354 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:53,355 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,355 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:53,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:53,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:53,355 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:53,355 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:53,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:53,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:53,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:53,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:53,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:53,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e41e8b8[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:53,356 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:53,356 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:53,357 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:53,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:53,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:53,357 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:53,357 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:53,357 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:53,357 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,357 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:53,358 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,358 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:53,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:53,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:53,358 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:53,358 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:53,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:53,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:53,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:53,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:53,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:53,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:53,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:53,361 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:53,361 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:53,361 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:53,361 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:53,361 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:53,362 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:53,362 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:53,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:53,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:53,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:53,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:53,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:53,362 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:53,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:53,362 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:53,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:53,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:53,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:53,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:53,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:53,363 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:53,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:53,365 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:53,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,365 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:53,365 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:53,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:53,365 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:53,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:53,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:53,365 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:53,365 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:53,365 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:53,365 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:53,365 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,366 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:53,366 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:53,366 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:53,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:53,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:53,366 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:53,440 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:53,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54262 deadline: 1681995413438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,442 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:53,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47054 deadline: 1681995413438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,442 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:53,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54244 deadline: 1681995413438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,466 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:53,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:53,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:53,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:53,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:53,468 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:53,468 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:53,468 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:53,468 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:53,468 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:53,468 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:53,469 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,469 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:53,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:53,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:53,469 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,469 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:53,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:53,469 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:53,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:53,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:53,470 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:53,470 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:53,470 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:53,470 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:53,470 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:53,470 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:53,470 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:53,470 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:53,470 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:53,470 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:53,470 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:53,470 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:53,470 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:53,471 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:53,471 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,471 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:53,471 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:53,471 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:53,471 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:53,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:53,472 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:53,472 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:53,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:53,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:53,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:53,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:53,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:53,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45904b2d[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:53,473 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:53,473 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:53,473 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:53,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:53,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:53,473 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:53,473 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:53,473 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:53,473 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,474 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:53,474 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,474 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:53,474 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:53,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:53,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:53,474 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:53,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:53,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:53,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:53,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:53,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:53,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:53,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:53,477 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:53,477 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:53,477 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:53,477 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:53,477 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:53,478 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:53,478 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:53,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:53,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:53,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:53,478 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:53,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:53,478 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:53,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:53,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:53,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:53,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:53,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:53,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:53,479 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:53,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:53,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:53,482 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:53,482 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:53,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:53,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:53,482 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,482 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:53,482 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:53,482 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:53,482 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:53,483 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:53,483 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:53,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:53,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:53,482 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,483 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:53,483 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:53,483 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:53,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:53,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:53,583 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:53,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:53,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:53,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:53,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:53,585 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:53,585 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:53,585 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:53,585 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:53,586 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:53,586 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:53,586 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:53,587 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:53,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:53,587 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:53,587 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:53,587 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:53,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:53,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:53,587 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:53,588 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:53,588 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:53,588 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:53,588 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:53,588 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:53,588 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:53,588 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:53,588 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:53,588 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:53,588 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:53,588 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:53,588 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:53,588 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:53,588 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,588 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:53,588 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:53,589 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,589 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:53,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:53,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:53,589 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:53,589 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:53,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:53,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:53,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:53,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:53,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:53,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1dfa3e14[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:53,590 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:53,590 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:53,591 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:53,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:53,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:53,591 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:53,591 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:53,591 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:53,591 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,591 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:53,592 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:53,592 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:53,592 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,592 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:53,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:53,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:53,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:53,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:53,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:53,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:53,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:53,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:53,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:53,594 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:53,594 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:53,594 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:53,594 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:53,594 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:53,595 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:53,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:53,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:53,595 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:53,595 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:53,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:53,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:53,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:53,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:53,595 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:53,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:53,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:53,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:53,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:53,596 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:53,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:53,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:53,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,598 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:53,598 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:53,598 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:53,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:53,598 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:53,598 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:53,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:53,598 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:53,598 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:53,598 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:53,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:53,598 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,598 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:53,598 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:53,598 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:53,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:53,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:53,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:53,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:53,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54262 deadline: 1681995413641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,645 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:53,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54244 deadline: 1681995413643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,645 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:53,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47054 deadline: 1681995413644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,698 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:53,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:53,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:53,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:53,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:53,701 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:53,701 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:53,701 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:53,701 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:53,701 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:53,701 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:53,702 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,702 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:53,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:53,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:53,702 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,702 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:53,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:53,702 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:53,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:53,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:53,703 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:53,703 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:53,703 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:53,703 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:53,703 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:53,703 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:53,703 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:53,703 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:53,703 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:53,703 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:53,703 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:53,703 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:53,703 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:53,703 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:53,704 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,704 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:53,704 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:53,704 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,704 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:53,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:53,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:53,704 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:53,704 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:53,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:53,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:53,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:53,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:53,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:53,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c05b4b0[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:53,706 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:53,706 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:53,706 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:53,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:53,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:53,706 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:53,706 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:53,706 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,706 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:53,706 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:53,707 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,707 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:53,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:53,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:53,707 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:53,707 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:53,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:53,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:53,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:53,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:53,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:53,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:53,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:53,710 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:53,711 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:53,711 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:53,711 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:53,711 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:53,711 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:53,711 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:53,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:53,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:53,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:53,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:53,711 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:53,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:53,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:53,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:53,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:53,712 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:53,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:53,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:53,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:53,713 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:53,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:53,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,715 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,715 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:53,715 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:53,715 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:53,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:53,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:53,715 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:53,715 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:53,715 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,715 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:53,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:53,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:53,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:53,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:53,715 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:53,715 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:53,715 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:53,715 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:53,716 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:53,741 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=340 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/2459d714e30940d5a71b84b0c22b57f6 2023-04-20 12:55:53,744 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/9b4b7ac06f9346bea6e3a44142aefb6b as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/9b4b7ac06f9346bea6e3a44142aefb6b 2023-04-20 12:55:53,748 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/9b4b7ac06f9346bea6e3a44142aefb6b, entries=200, sequenceid=340, filesize=14.3 K 2023-04-20 12:55:53,749 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/8575608848b341f398ce7b5adea8980a as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/8575608848b341f398ce7b5adea8980a 2023-04-20 12:55:53,753 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/8575608848b341f398ce7b5adea8980a, entries=150, sequenceid=340, filesize=11.9 K 2023-04-20 12:55:53,754 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/2459d714e30940d5a71b84b0c22b57f6 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/2459d714e30940d5a71b84b0c22b57f6 2023-04-20 12:55:53,758 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/2459d714e30940d5a71b84b0c22b57f6, entries=150, sequenceid=340, filesize=11.9 K 2023-04-20 12:55:53,759 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for bed4cc4c061ff4314d49ea7add6e3672 in 448ms, sequenceid=340, compaction requested=true 2023-04-20 12:55:53,759 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:55:53,759 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:55:53,759 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:55:53,759 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:55:53,759 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:55:53,759 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:55:53,761 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39646 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:55:53,761 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): bed4cc4c061ff4314d49ea7add6e3672/A is initiating minor compaction (all files) 2023-04-20 12:55:53,761 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bed4cc4c061ff4314d49ea7add6e3672/A in TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:53,761 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/c952d98f537e4d9aa11dd71394d87f50, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/e6a756def45a438c8751f0461bb2612d, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/9b4b7ac06f9346bea6e3a44142aefb6b] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp, totalSize=38.7 K 2023-04-20 12:55:53,761 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:55:53,761 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): bed4cc4c061ff4314d49ea7add6e3672/B is initiating minor compaction (all files) 2023-04-20 12:55:53,761 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bed4cc4c061ff4314d49ea7add6e3672/B in TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:53,761 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/1f7bd7186f4f453e9a203b651a3e4dc8, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/5d2c401500a141ee83820384ad2be706, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/8575608848b341f398ce7b5adea8980a] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp, totalSize=36.3 K 2023-04-20 12:55:53,762 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting c952d98f537e4d9aa11dd71394d87f50, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1681995350688 2023-04-20 12:55:53,762 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 1f7bd7186f4f453e9a203b651a3e4dc8, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1681995350688 2023-04-20 12:55:53,763 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting e6a756def45a438c8751f0461bb2612d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1681995351029 2023-04-20 12:55:53,763 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 5d2c401500a141ee83820384ad2be706, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1681995351029 2023-04-20 12:55:53,763 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 9b4b7ac06f9346bea6e3a44142aefb6b, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1681995352186 2023-04-20 12:55:53,763 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 8575608848b341f398ce7b5adea8980a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1681995352186 2023-04-20 12:55:53,769 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): bed4cc4c061ff4314d49ea7add6e3672#A#compaction#445 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:55:53,769 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): bed4cc4c061ff4314d49ea7add6e3672#B#compaction#446 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:55:53,804 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/e751512dda3141aea4db4d9279ac5c43 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/e751512dda3141aea4db4d9279ac5c43 2023-04-20 12:55:53,808 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bed4cc4c061ff4314d49ea7add6e3672/A of bed4cc4c061ff4314d49ea7add6e3672 into e751512dda3141aea4db4d9279ac5c43(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:55:53,808 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:55:53,808 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672., storeName=bed4cc4c061ff4314d49ea7add6e3672/A, priority=13, startTime=1681995353759; duration=0sec 2023-04-20 12:55:53,808 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:55:53,808 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:55:53,809 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:55:53,809 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): bed4cc4c061ff4314d49ea7add6e3672/C is initiating minor compaction (all files) 2023-04-20 12:55:53,809 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bed4cc4c061ff4314d49ea7add6e3672/C in TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:53,809 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/8b9a4a865144454b964ffc775dd2889e, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/12e870813b964fab963bd7b327c4361d, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/2459d714e30940d5a71b84b0c22b57f6] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp, totalSize=36.3 K 2023-04-20 12:55:53,810 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 8b9a4a865144454b964ffc775dd2889e, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1681995350688 2023-04-20 12:55:53,810 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 12e870813b964fab963bd7b327c4361d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1681995351029 2023-04-20 12:55:53,810 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 2459d714e30940d5a71b84b0c22b57f6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1681995352186 2023-04-20 12:55:53,816 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): bed4cc4c061ff4314d49ea7add6e3672#C#compaction#447 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:55:53,816 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:53,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:53,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:53,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:53,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:53,822 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:53,822 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:53,822 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:53,822 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:53,822 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:53,822 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:53,823 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,823 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:53,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:53,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:53,825 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,825 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:53,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:53,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:53,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:53,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:53,825 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:53,826 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:53,826 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:53,826 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:53,826 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:53,826 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:53,826 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:53,826 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:53,826 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:53,826 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:53,826 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bed4cc4c061ff4314d49ea7add6e3672 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-20 12:55:53,826 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=A 2023-04-20 12:55:53,826 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:53,826 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=B 2023-04-20 12:55:53,826 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:53,826 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=C 2023-04-20 12:55:53,826 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:53,833 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=353 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/88c6ab2656d94fb18d0b30581d9bd45f 2023-04-20 12:55:53,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on bed4cc4c061ff4314d49ea7add6e3672 2023-04-20 12:55:53,949 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:53,985 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:53,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54244 deadline: 1681995413979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,989 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:53,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54262 deadline: 1681995413982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:53,989 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:53,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47054 deadline: 1681995413985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:54,089 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:54,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54244 deadline: 1681995414086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:54,092 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:54,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47054 deadline: 1681995414090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:54,092 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:54,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54262 deadline: 1681995414090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:54,204 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/bad6ee59968948f5bc17410fd689c35b as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/bad6ee59968948f5bc17410fd689c35b 2023-04-20 12:55:54,207 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bed4cc4c061ff4314d49ea7add6e3672/B of bed4cc4c061ff4314d49ea7add6e3672 into bad6ee59968948f5bc17410fd689c35b(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:55:54,207 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:55:54,207 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672., storeName=bed4cc4c061ff4314d49ea7add6e3672/B, priority=13, startTime=1681995353759; duration=0sec 2023-04-20 12:55:54,207 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:55:54,227 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/b10dab1e4b5d48178a1dcd76f2330e2e as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/b10dab1e4b5d48178a1dcd76f2330e2e 2023-04-20 12:55:54,230 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in bed4cc4c061ff4314d49ea7add6e3672/C of bed4cc4c061ff4314d49ea7add6e3672 into b10dab1e4b5d48178a1dcd76f2330e2e(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:55:54,230 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:55:54,230 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672., storeName=bed4cc4c061ff4314d49ea7add6e3672/C, priority=13, startTime=1681995353759; duration=0sec 2023-04-20 12:55:54,230 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:55:54,244 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=353 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/8104f4660b5f418cb131c5ba752086a1 2023-04-20 12:55:54,293 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:54,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54244 deadline: 1681995414291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:54,296 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:54,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47054 deadline: 1681995414293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:54,296 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:54,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54262 deadline: 1681995414294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:54,597 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:54,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54244 deadline: 1681995414596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:54,597 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:54,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54262 deadline: 1681995414597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:54,601 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:54,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47054 deadline: 1681995414598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:54,654 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=353 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/9c9b67e893a2454d86e7a80aae0104eb 2023-04-20 12:55:54,657 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/88c6ab2656d94fb18d0b30581d9bd45f as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/88c6ab2656d94fb18d0b30581d9bd45f 2023-04-20 12:55:54,666 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/88c6ab2656d94fb18d0b30581d9bd45f, entries=150, sequenceid=353, filesize=11.9 K 2023-04-20 12:55:54,667 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/8104f4660b5f418cb131c5ba752086a1 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/8104f4660b5f418cb131c5ba752086a1 2023-04-20 12:55:54,669 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/8104f4660b5f418cb131c5ba752086a1, entries=150, sequenceid=353, filesize=11.9 K 2023-04-20 12:55:54,670 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/9c9b67e893a2454d86e7a80aae0104eb as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/9c9b67e893a2454d86e7a80aae0104eb 2023-04-20 12:55:54,672 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/9c9b67e893a2454d86e7a80aae0104eb, entries=150, sequenceid=353, filesize=11.9 K 2023-04-20 12:55:54,673 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for bed4cc4c061ff4314d49ea7add6e3672 in 847ms, sequenceid=353, compaction requested=false 2023-04-20 12:55:54,673 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:55:54,673 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:54,673 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:54,673 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:54,673 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:54,673 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:54,673 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:54,674 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:54,674 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:54,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:54,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:54,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:54,674 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:54,674 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:54,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:54,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:54,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:54,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:54,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:54,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:54,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2db08ba0[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:54,675 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:54,675 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:54,676 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:54,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:54,676 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:54,676 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:54,676 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:54,676 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:54,676 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:54,677 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:54,677 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:54,677 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:54,677 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:54,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:54,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:54,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:54,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:54,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:54,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:54,678 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:54,679 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:54,679 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:54,679 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:54,679 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:54,679 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:54,679 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:54,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:54,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:54,679 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:54,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:54,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:54,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:54,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:54,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:54,680 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:54,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:54,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:54,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:54,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:54,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:54,681 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:54,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:54,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:54,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:54,682 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:54,682 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:54,682 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:54,682 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:54,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:54,682 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:54,682 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:54,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:54,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:54,682 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:54,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:54,682 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:54,683 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:54,682 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:54,683 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:54,683 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:54,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:54,783 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:54,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:54,784 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:54,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:54,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:54,785 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:54,785 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:54,785 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:54,785 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:54,786 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:54,786 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:54,786 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:54,786 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:54,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:54,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:54,786 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:54,786 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:54,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:54,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:54,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:54,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:54,787 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:54,787 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:54,787 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:54,787 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:54,787 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:54,787 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:54,787 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:54,788 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:54,788 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:54,788 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:54,788 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bed4cc4c061ff4314d49ea7add6e3672 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-20 12:55:54,788 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=A 2023-04-20 12:55:54,788 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:54,788 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=B 2023-04-20 12:55:54,788 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:54,788 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=C 2023-04-20 12:55:54,788 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:54,798 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=375 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/7a5062d3dc9a4d98abe411d30b45ded3 2023-04-20 12:55:55,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on bed4cc4c061ff4314d49ea7add6e3672 2023-04-20 12:55:55,100 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:55,123 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:55,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47054 deadline: 1681995415118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,124 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:55,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54262 deadline: 1681995415119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,125 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:55,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54244 deadline: 1681995415119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,205 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=375 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/f9810e2b4588428389d78bbf0e1dcc8b 2023-04-20 12:55:55,227 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:55,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47054 deadline: 1681995415224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,228 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:55,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54262 deadline: 1681995415225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,229 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:55,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54244 deadline: 1681995415225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,435 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:55,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47054 deadline: 1681995415428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,435 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:55,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54262 deadline: 1681995415430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,435 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:55,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54244 deadline: 1681995415430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,613 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=375 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/8eb784835dae4672b0c8c7a943ee1e32 2023-04-20 12:55:55,616 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/7a5062d3dc9a4d98abe411d30b45ded3 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/7a5062d3dc9a4d98abe411d30b45ded3 2023-04-20 12:55:55,619 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/7a5062d3dc9a4d98abe411d30b45ded3, entries=150, sequenceid=375, filesize=11.9 K 2023-04-20 12:55:55,619 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/f9810e2b4588428389d78bbf0e1dcc8b as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/f9810e2b4588428389d78bbf0e1dcc8b 2023-04-20 12:55:55,622 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/f9810e2b4588428389d78bbf0e1dcc8b, entries=150, sequenceid=375, filesize=11.9 K 2023-04-20 12:55:55,622 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/8eb784835dae4672b0c8c7a943ee1e32 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/8eb784835dae4672b0c8c7a943ee1e32 2023-04-20 12:55:55,625 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/8eb784835dae4672b0c8c7a943ee1e32, entries=150, sequenceid=375, filesize=11.9 K 2023-04-20 12:55:55,626 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for bed4cc4c061ff4314d49ea7add6e3672 in 838ms, sequenceid=375, compaction requested=true 2023-04-20 12:55:55,626 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:55:55,626 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:55,626 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:55,626 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:55,626 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,626 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:55,626 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:55,627 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,627 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:55,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:55,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:55,627 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:55,627 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:55,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:55,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:55,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:55,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:55,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:55,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a3e6418[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:55,629 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:55,629 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:55,629 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:55,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:55,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:55,630 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,630 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:55,633 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:55,633 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:55,633 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:55,636 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:55,636 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,636 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:55,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:55,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:55,636 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:55,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:55,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:55,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:55,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:55,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:55,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:55,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:55,638 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:55,638 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:55,638 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:55,638 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:55,638 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:55,640 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:55,640 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:55,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:55,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:55,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:55,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:55,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:55,640 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:55,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:55,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:55,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:55,641 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:55,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:55,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:55,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:55,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,642 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:55,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:55,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,643 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,643 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:55,643 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:55,643 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,643 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:55,643 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:55,643 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:55,643 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:55,643 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:55,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:55,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:55,644 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:55,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:55,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:55,644 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:55,644 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:55,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:55,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:55,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on bed4cc4c061ff4314d49ea7add6e3672 2023-04-20 12:55:55,744 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bed4cc4c061ff4314d49ea7add6e3672 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-20 12:55:55,744 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:55,744 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=A 2023-04-20 12:55:55,744 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:55,744 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=B 2023-04-20 12:55:55,745 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:55,745 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=C 2023-04-20 12:55:55,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:55,745 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:55,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:55,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:55,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:55,747 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:55,747 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:55,747 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:55,747 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:55,747 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:55,747 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:55,748 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,748 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:55,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:55,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:55,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:55,748 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,748 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:55,748 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:55,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:55,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:55,749 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:55,749 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:55,749 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:55,749 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:55,749 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:55,749 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:55,749 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:55,749 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:55,749 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:55,749 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:55,749 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:55,749 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:55,749 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:55,749 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:55,749 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,749 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:55,750 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:55,750 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:55,750 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:55,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:55,750 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=389 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/ce641c767154433aa4e3780f3d8212d4 2023-04-20 12:55:55,751 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:55,751 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:55,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:55,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:55,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:55,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:55,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:55,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26f8fdd5[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:55,752 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:55,752 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:55,753 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:55,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:55,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:55,753 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:55,753 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:55,753 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:55,753 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,753 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:55,754 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,754 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:55,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:55,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:55,754 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:55,754 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:55,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:55,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:55,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:55,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:55,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:55,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:55,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:55,756 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:55,756 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:55,756 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:55,756 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:55,756 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:55,757 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:55,757 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:55,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:55,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:55,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:55,757 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:55,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:55,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:55,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:55,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:55,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:55,758 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:55,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:55,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:55,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:55,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,759 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:55,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:55,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,762 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,762 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:55,762 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:55,762 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:55,762 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:55,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:55,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:55,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:55,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:55,762 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:55,762 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:55,762 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,762 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:55,762 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:55,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:55,762 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:55,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:55,762 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:55,780 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:55,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54262 deadline: 1681995415774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,780 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:55,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54244 deadline: 1681995415774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,781 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:55,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47054 deadline: 1681995415775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,862 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:55,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:55,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:55,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:55,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:55,865 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:55,865 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:55,865 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:55,865 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:55,865 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:55,865 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:55,867 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:55,867 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:55,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:55,867 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,867 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:55,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:55,868 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:55,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:55,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:55,868 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:55,868 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:55,869 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:55,869 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:55,869 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:55,869 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:55,869 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:55,869 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:55,869 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:55,869 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:55,869 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:55,869 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:55,869 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:55,869 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:55,869 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,869 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:55,869 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:55,870 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,870 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:55,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:55,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:55,870 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:55,870 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:55,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:55,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:55,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:55,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:55,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:55,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17e97d8e[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:55,871 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:55,871 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:55,872 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:55,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:55,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:55,872 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,872 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:55,872 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:55,872 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:55,872 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:55,873 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,873 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:55,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:55,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:55,873 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:55,873 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:55,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:55,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:55,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:55,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:55,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:55,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:55,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:55,875 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:55,875 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:55,875 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:55,875 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:55,875 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:55,876 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:55,876 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:55,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:55,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:55,876 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:55,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:55,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:55,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:55,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:55,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:55,876 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:55,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:55,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:55,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:55,877 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:55,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:55,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:55,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,878 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,878 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:55,878 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:55,878 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:55,878 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:55,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:55,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:55,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:55,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:55,878 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:55,878 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,878 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:55,879 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:55,879 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:55,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:55,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:55,879 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:55,879 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:55,879 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:55,881 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:55,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54262 deadline: 1681995415881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,888 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:55,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54244 deadline: 1681995415882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,889 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:55,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47054 deadline: 1681995415882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,979 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:55,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:55,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:55,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:55,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:55,982 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:55,982 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:55,982 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:55,982 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:55,983 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:55,983 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:55,984 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,984 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:55,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:55,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:55,984 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,984 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:55,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:55,984 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:55,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:55,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:55,985 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:55,985 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:55,985 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:55,985 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:55,985 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:55,985 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:55,985 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:55,985 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:55,985 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:55,985 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:55,985 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:55,985 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:55,985 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:55,985 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:55,985 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,985 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:55,985 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:55,986 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:55,986 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:55,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:55,986 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:55,986 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:55,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:55,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:55,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:55,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:55,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:55,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ead1958[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:55,988 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:55,988 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:55,988 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:55,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:55,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:55,988 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:55,989 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:55,989 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:55,989 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,989 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:55,989 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,989 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:55,989 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:55,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:55,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:55,989 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:55,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:55,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:55,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:55,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:55,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:55,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:55,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:55,991 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:55,991 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:55,991 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:55,991 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:55,991 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:55,992 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:55,992 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:55,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:55,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:55,992 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:55,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:55,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:55,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:55,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:55,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:55,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:55,993 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:55,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:55,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:55,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:55,994 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:55,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:55,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:55,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,006 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,006 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:56,006 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:56,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:56,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:56,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:56,006 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:56,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:56,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:56,006 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:56,007 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:56,007 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,007 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:56,007 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:56,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:56,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:56,007 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:56,007 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:56,007 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:56,086 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:56,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54262 deadline: 1681995416082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,093 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:56,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54244 deadline: 1681995416089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,093 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:56,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47054 deadline: 1681995416089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,107 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:56,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:56,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:56,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:56,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:56,110 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:56,110 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:56,110 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:56,110 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:56,110 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:56,110 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:56,111 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,111 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:56,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:56,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:56,111 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:56,111 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:56,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:56,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:56,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:56,112 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:56,112 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:56,112 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:56,112 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:56,112 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:56,112 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:56,112 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:56,112 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:56,112 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:56,112 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:56,112 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:56,112 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:56,112 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:56,112 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:56,112 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,113 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:56,113 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:56,113 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,113 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:56,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:56,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:56,114 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:56,114 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:56,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:56,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:56,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:56,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:56,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:56,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@272ab431[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:56,115 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:56,115 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:56,115 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:56,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:56,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:56,115 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:56,115 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:56,115 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:56,115 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,115 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:56,116 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,116 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:56,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:56,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:56,116 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:56,116 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:56,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:56,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:56,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:56,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:56,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:56,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:56,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:56,118 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:56,118 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:56,118 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:56,118 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:56,118 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:56,119 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:56,119 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:56,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:56,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:56,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:56,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:56,119 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:56,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:56,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:56,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:56,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:56,119 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:56,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:56,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:56,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:56,120 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:56,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:56,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,121 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,121 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:56,121 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:56,121 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,121 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:56,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:56,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:56,121 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:56,121 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:56,121 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:56,121 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:56,121 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:56,122 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:56,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:56,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:56,121 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:56,122 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:56,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:56,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:56,160 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=389 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/5f745d7eae6241a58fb42adb39daa4f2 2023-04-20 12:55:56,222 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:56,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:56,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:56,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:56,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:56,224 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:56,224 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:56,224 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:56,224 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:56,225 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:56,225 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:56,225 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:56,226 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:56,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:56,226 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,226 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:56,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:56,226 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:56,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:56,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:56,226 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:56,226 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:56,226 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:56,227 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:56,227 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:56,227 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:56,227 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:56,227 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:56,227 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:56,227 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:56,227 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:56,227 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:56,227 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:56,227 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:56,227 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,227 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:56,227 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:56,228 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:56,228 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:56,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:56,228 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:56,228 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:56,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:56,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:56,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:56,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:56,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:56,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@465df354[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:56,229 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:56,229 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:56,230 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:56,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:56,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:56,230 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:56,230 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:56,230 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,230 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:56,230 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:56,231 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:56,231 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:56,231 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,231 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:56,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:56,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:56,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:56,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:56,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:56,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:56,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:56,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:56,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:56,233 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:56,233 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:56,233 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:56,233 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:56,233 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:56,233 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:56,233 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:56,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:56,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:56,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:56,233 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:56,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:56,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:56,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:56,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:56,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:56,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:56,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:56,234 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:56,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:56,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:56,235 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:56,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,237 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:56,237 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:56,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:56,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:56,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:56,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:56,237 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:56,237 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,237 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:56,237 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:56,237 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:56,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:56,238 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:56,237 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:56,238 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:56,238 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:56,238 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:56,338 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:56,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:56,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:56,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:56,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:56,340 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:56,340 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:56,340 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:56,340 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:56,340 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:56,340 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:56,341 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:56,341 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:56,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:56,341 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,341 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:56,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:56,341 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:56,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:56,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:56,342 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:56,342 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:56,342 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:56,342 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:56,342 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:56,342 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:56,342 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:56,342 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:56,342 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:56,342 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:56,342 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:56,342 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:56,343 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:56,343 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:56,343 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,343 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:56,343 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:56,343 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,343 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:56,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:56,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:56,344 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:56,344 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:56,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:56,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:56,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:56,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:56,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:56,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a04f31c[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:56,345 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:56,345 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:56,345 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:56,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:56,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:56,345 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:56,345 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:56,345 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:56,345 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,345 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:56,346 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:56,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:56,346 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:56,346 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:56,346 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:56,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:56,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:56,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:56,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:56,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:56,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:56,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:56,348 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:56,348 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:56,348 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:56,348 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:56,348 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:56,349 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:56,349 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:56,349 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:56,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:56,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:56,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:56,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:56,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:56,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:56,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:56,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:56,349 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:56,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:56,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:56,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:56,350 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:56,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:56,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,351 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,351 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:56,351 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:56,352 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:56,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:56,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:56,352 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,351 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:56,352 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:56,352 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:56,352 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:56,352 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:56,352 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:56,352 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:56,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:56,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:56,352 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:56,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:56,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:56,395 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:56,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54262 deadline: 1681995416389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,396 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:56,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47054 deadline: 1681995416394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,396 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:56,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54244 deadline: 1681995416395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,452 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:56,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:56,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:56,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:56,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:56,455 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:56,455 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:56,455 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:56,455 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:56,455 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:56,455 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:56,456 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:56,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:56,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:56,456 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:56,456 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,456 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:56,456 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:56,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:56,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:56,457 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:56,457 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:56,457 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:56,457 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:56,457 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:56,457 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:56,457 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:56,457 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:56,457 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:56,457 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:56,457 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:56,457 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:56,457 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:56,457 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:56,457 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,458 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:56,458 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:56,458 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,458 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:56,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:56,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:56,459 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:56,459 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:56,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:56,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:56,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:56,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:56,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:56,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4cb54edd[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:56,460 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:56,460 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:56,460 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:56,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:56,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:56,460 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:56,460 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:56,460 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,460 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:56,460 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:56,461 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,461 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:56,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:56,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:56,461 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:56,461 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:56,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:56,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:56,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:56,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:56,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:56,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:56,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:56,463 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:56,463 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:56,463 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:56,463 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:56,463 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:56,465 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:56,465 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:56,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:56,465 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:56,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:56,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:56,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:56,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:56,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:56,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:56,465 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:56,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:56,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:56,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:56,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:56,466 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:56,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:56,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,467 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,467 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:56,467 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:56,468 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:56,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:56,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:56,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:56,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:56,468 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:56,468 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:56,468 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:56,468 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:56,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:56,468 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:56,468 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:56,468 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:56,468 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:56,468 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:56,568 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:56,569 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=389 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/90f1c8f37327410e9f06a588e4820955 2023-04-20 12:55:56,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:56,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:56,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:56,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:56,573 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:56,573 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:56,573 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:56,573 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:56,573 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/ce641c767154433aa4e3780f3d8212d4 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/ce641c767154433aa4e3780f3d8212d4 2023-04-20 12:55:56,573 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:56,573 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:56,574 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:56,574 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:56,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:56,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:56,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:56,575 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,575 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:56,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:56,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:56,576 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:56,576 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:56,576 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:56,576 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:56,576 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:56,576 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:56,577 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:56,577 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:56,577 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:56,577 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:56,577 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:56,577 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:56,577 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:56,577 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:56,577 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,577 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:56,577 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:56,578 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/ce641c767154433aa4e3780f3d8212d4, entries=200, sequenceid=389, filesize=14.3 K 2023-04-20 12:55:56,579 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/5f745d7eae6241a58fb42adb39daa4f2 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/5f745d7eae6241a58fb42adb39daa4f2 2023-04-20 12:55:56,581 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:56,581 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:56,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:56,582 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:56,582 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:56,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:56,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:56,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:56,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:56,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:56,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e4ce046[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:56,584 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:56,584 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:56,584 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/5f745d7eae6241a58fb42adb39daa4f2, entries=150, sequenceid=389, filesize=11.9 K 2023-04-20 12:55:56,585 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:56,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:56,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:56,585 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:56,585 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:56,585 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:56,586 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/90f1c8f37327410e9f06a588e4820955 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/90f1c8f37327410e9f06a588e4820955 2023-04-20 12:55:56,586 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,586 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:56,588 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,588 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:56,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:56,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:56,588 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:56,588 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:56,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:56,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:56,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:56,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:56,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:56,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:56,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:56,591 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:56,592 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:56,592 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:56,592 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:56,592 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:56,591 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/90f1c8f37327410e9f06a588e4820955, entries=150, sequenceid=389, filesize=11.9 K 2023-04-20 12:55:56,594 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for bed4cc4c061ff4314d49ea7add6e3672 in 850ms, sequenceid=389, compaction requested=true 2023-04-20 12:55:56,594 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:55:56,594 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:55:56,594 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:55:56,594 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 12:55:56,594 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 12:55:56,594 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:55:56,596 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 12:55:56,596 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51934 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 12:55:56,596 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): bed4cc4c061ff4314d49ea7add6e3672/B is initiating minor compaction (all files) 2023-04-20 12:55:56,596 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): bed4cc4c061ff4314d49ea7add6e3672/A is initiating minor compaction (all files) 2023-04-20 12:55:56,597 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bed4cc4c061ff4314d49ea7add6e3672/B in TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:56,597 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bed4cc4c061ff4314d49ea7add6e3672/A in TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:56,597 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/e751512dda3141aea4db4d9279ac5c43, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/88c6ab2656d94fb18d0b30581d9bd45f, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/7a5062d3dc9a4d98abe411d30b45ded3, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/ce641c767154433aa4e3780f3d8212d4] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp, totalSize=50.7 K 2023-04-20 12:55:56,597 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/bad6ee59968948f5bc17410fd689c35b, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/8104f4660b5f418cb131c5ba752086a1, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/f9810e2b4588428389d78bbf0e1dcc8b, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/5f745d7eae6241a58fb42adb39daa4f2] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp, totalSize=48.3 K 2023-04-20 12:55:56,597 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting e751512dda3141aea4db4d9279ac5c43, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1681995352186 2023-04-20 12:55:56,597 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting bad6ee59968948f5bc17410fd689c35b, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1681995352186 2023-04-20 12:55:56,597 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 88c6ab2656d94fb18d0b30581d9bd45f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=353, earliestPutTs=1681995353330 2023-04-20 12:55:56,597 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 8104f4660b5f418cb131c5ba752086a1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=353, earliestPutTs=1681995353330 2023-04-20 12:55:56,598 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 7a5062d3dc9a4d98abe411d30b45ded3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1681995353970 2023-04-20 12:55:56,598 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting f9810e2b4588428389d78bbf0e1dcc8b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1681995353970 2023-04-20 12:55:56,598 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting ce641c767154433aa4e3780f3d8212d4, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=389, earliestPutTs=1681995355116 2023-04-20 12:55:56,598 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:56,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:56,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:56,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:56,598 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:56,598 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:56,598 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 5f745d7eae6241a58fb42adb39daa4f2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=389, earliestPutTs=1681995355116 2023-04-20 12:55:56,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:56,599 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:56,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:56,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:56,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,601 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:56,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:56,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:56,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:56,607 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:56,607 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:56,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:56,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:56,607 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): bed4cc4c061ff4314d49ea7add6e3672#A#compaction#457 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:55:56,608 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:56,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:56,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:56,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:56,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:56,608 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,608 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:56,608 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:56,608 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,608 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:56,608 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:56,608 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:56,608 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:56,608 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:56,611 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): bed4cc4c061ff4314d49ea7add6e3672#B#compaction#458 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:55:56,635 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/a35cd88c88514174b48d7c26998ad1de as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/a35cd88c88514174b48d7c26998ad1de 2023-04-20 12:55:56,638 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bed4cc4c061ff4314d49ea7add6e3672/B of bed4cc4c061ff4314d49ea7add6e3672 into a35cd88c88514174b48d7c26998ad1de(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:55:56,638 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:55:56,638 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672., storeName=bed4cc4c061ff4314d49ea7add6e3672/B, priority=12, startTime=1681995356594; duration=0sec 2023-04-20 12:55:56,638 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:55:56,638 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-20 12:55:56,640 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-20 12:55:56,641 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): bed4cc4c061ff4314d49ea7add6e3672/C is initiating minor compaction (all files) 2023-04-20 12:55:56,641 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bed4cc4c061ff4314d49ea7add6e3672/C in TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:56,641 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/b10dab1e4b5d48178a1dcd76f2330e2e, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/9c9b67e893a2454d86e7a80aae0104eb, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/8eb784835dae4672b0c8c7a943ee1e32, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/90f1c8f37327410e9f06a588e4820955] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp, totalSize=48.3 K 2023-04-20 12:55:56,641 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting b10dab1e4b5d48178a1dcd76f2330e2e, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1681995352186 2023-04-20 12:55:56,641 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 9c9b67e893a2454d86e7a80aae0104eb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=353, earliestPutTs=1681995353330 2023-04-20 12:55:56,642 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 8eb784835dae4672b0c8c7a943ee1e32, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1681995353970 2023-04-20 12:55:56,642 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 90f1c8f37327410e9f06a588e4820955, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=389, earliestPutTs=1681995355116 2023-04-20 12:55:56,652 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): bed4cc4c061ff4314d49ea7add6e3672#C#compaction#459 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:55:56,708 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:56,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:56,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:56,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:56,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:56,711 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:56,711 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:56,711 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:56,711 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:56,711 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:56,712 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:56,712 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,712 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:56,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:56,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:56,712 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,712 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:56,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:56,713 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:56,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:56,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:56,713 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:56,713 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:56,713 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:56,713 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:56,713 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:56,713 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:56,713 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:56,713 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:56,713 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:56,713 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:56,714 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bed4cc4c061ff4314d49ea7add6e3672 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-20 12:55:56,714 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=A 2023-04-20 12:55:56,714 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:56,714 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=B 2023-04-20 12:55:56,714 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:56,714 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=C 2023-04-20 12:55:56,714 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:56,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on bed4cc4c061ff4314d49ea7add6e3672 2023-04-20 12:55:56,900 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:56,920 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:56,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47054 deadline: 1681995416915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,920 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:56,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54262 deadline: 1681995416916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:56,922 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:56,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54244 deadline: 1681995416916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:57,028 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:57,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47054 deadline: 1681995417021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:57,028 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:57,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54262 deadline: 1681995417021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:57,029 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:57,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54244 deadline: 1681995417022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:57,033 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/4ff21af4bfae4c4785eb6f97b8603295 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/4ff21af4bfae4c4785eb6f97b8603295 2023-04-20 12:55:57,036 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bed4cc4c061ff4314d49ea7add6e3672/A of bed4cc4c061ff4314d49ea7add6e3672 into 4ff21af4bfae4c4785eb6f97b8603295(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:55:57,036 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:55:57,036 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672., storeName=bed4cc4c061ff4314d49ea7add6e3672/A, priority=12, startTime=1681995356594; duration=0sec 2023-04-20 12:55:57,036 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:55:57,064 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/f4cf31c96e714e979e444a023bd61206 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/f4cf31c96e714e979e444a023bd61206 2023-04-20 12:55:57,070 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in bed4cc4c061ff4314d49ea7add6e3672/C of bed4cc4c061ff4314d49ea7add6e3672 into f4cf31c96e714e979e444a023bd61206(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:55:57,070 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:55:57,070 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672., storeName=bed4cc4c061ff4314d49ea7add6e3672/C, priority=12, startTime=1681995356594; duration=0sec 2023-04-20 12:55:57,070 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:55:57,121 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=408 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/bc73d51254314ad4935c7e74720d796a 2023-04-20 12:55:57,231 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:57,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47054 deadline: 1681995417229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:57,232 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:57,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54262 deadline: 1681995417229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:57,232 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:57,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54244 deadline: 1681995417230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:57,534 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:57,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47054 deadline: 1681995417532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:57,536 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:57,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54262 deadline: 1681995417534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:57,536 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=408 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/cc440fccb4034f9eba49efe9a0738fc8 2023-04-20 12:55:57,536 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:57,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54244 deadline: 1681995417534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:57,544 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=408 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/3b40cd5f4bd547499063bd4723b019a8 2023-04-20 12:55:57,547 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/bc73d51254314ad4935c7e74720d796a as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/bc73d51254314ad4935c7e74720d796a 2023-04-20 12:55:57,551 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/bc73d51254314ad4935c7e74720d796a, entries=150, sequenceid=408, filesize=11.9 K 2023-04-20 12:55:57,551 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/cc440fccb4034f9eba49efe9a0738fc8 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/cc440fccb4034f9eba49efe9a0738fc8 2023-04-20 12:55:57,554 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/cc440fccb4034f9eba49efe9a0738fc8, entries=150, sequenceid=408, filesize=11.9 K 2023-04-20 12:55:57,554 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/3b40cd5f4bd547499063bd4723b019a8 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/3b40cd5f4bd547499063bd4723b019a8 2023-04-20 12:55:57,557 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/3b40cd5f4bd547499063bd4723b019a8, entries=150, sequenceid=408, filesize=11.9 K 2023-04-20 12:55:57,558 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for bed4cc4c061ff4314d49ea7add6e3672 in 843ms, sequenceid=408, compaction requested=false 2023-04-20 12:55:57,558 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:55:57,558 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:57,558 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:57,558 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:57,558 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:57,558 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:57,558 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:57,559 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:57,559 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:57,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:57,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:57,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:57,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:57,559 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:57,560 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:57,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:57,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:57,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:57,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:57,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:57,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17226388[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:57,561 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:57,561 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:57,562 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:57,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:57,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:57,562 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:57,562 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:57,562 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:57,563 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:57,563 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:57,563 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:57,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:57,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:57,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:57,563 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:57,563 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:57,564 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:57,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:57,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:57,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:57,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:57,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:57,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:57,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:57,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:57,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:57,566 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:57,566 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:57,566 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:57,566 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:57,566 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:57,577 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:57,577 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:57,577 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:57,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:57,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:57,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:57,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:57,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:57,578 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:57,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:57,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:57,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:57,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:57,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:57,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:57,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:57,590 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:57,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:57,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:57,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:57,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:57,592 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:57,592 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:57,592 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:57,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:57,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:57,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:57,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:57,592 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:57,592 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:57,592 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:57,593 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:57,593 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:57,593 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:57,593 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:57,593 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:57,593 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:57,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:57,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:57,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:57,693 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:57,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:57,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:57,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:57,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:57,696 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:57,696 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:57,696 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:57,696 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:57,696 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:57,696 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:57,697 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:57,697 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:57,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:57,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:57,697 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:57,697 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:57,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:57,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:57,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:57,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:57,698 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:57,699 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:57,699 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:57,699 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:57,699 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:57,699 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:57,699 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:57,699 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:57,699 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:57,699 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bed4cc4c061ff4314d49ea7add6e3672 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-20 12:55:57,699 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=A 2023-04-20 12:55:57,700 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:57,700 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=B 2023-04-20 12:55:57,700 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:57,700 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=C 2023-04-20 12:55:57,700 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:58,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on bed4cc4c061ff4314d49ea7add6e3672 2023-04-20 12:55:58,041 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:58,067 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:58,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54262 deadline: 1681995418065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:58,071 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:58,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47054 deadline: 1681995418066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:58,073 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:58,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54244 deadline: 1681995418067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:58,112 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=424 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/01f03213ceea4a0388285672079cb527 2023-04-20 12:55:58,172 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:58,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54262 deadline: 1681995418168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:58,176 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:58,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47054 deadline: 1681995418172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:58,178 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:58,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54244 deadline: 1681995418174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:58,376 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:58,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54262 deadline: 1681995418374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:58,381 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:58,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47054 deadline: 1681995418378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:58,381 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:58,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54244 deadline: 1681995418379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:58,523 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=424 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/5405bf40d349446dad18422f9ddf04f2 2023-04-20 12:55:58,680 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:58,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54262 deadline: 1681995418678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:58,685 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:58,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47054 deadline: 1681995418683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:58,685 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:58,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54244 deadline: 1681995418683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:58,940 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=424 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/747db884724b4ae986da46a6dc588d9c 2023-04-20 12:55:58,943 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/01f03213ceea4a0388285672079cb527 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/01f03213ceea4a0388285672079cb527 2023-04-20 12:55:58,947 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/01f03213ceea4a0388285672079cb527, entries=150, sequenceid=424, filesize=11.9 K 2023-04-20 12:55:58,948 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/5405bf40d349446dad18422f9ddf04f2 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/5405bf40d349446dad18422f9ddf04f2 2023-04-20 12:55:58,951 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/5405bf40d349446dad18422f9ddf04f2, entries=150, sequenceid=424, filesize=11.9 K 2023-04-20 12:55:58,951 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/747db884724b4ae986da46a6dc588d9c as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/747db884724b4ae986da46a6dc588d9c 2023-04-20 12:55:58,954 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/747db884724b4ae986da46a6dc588d9c, entries=150, sequenceid=424, filesize=11.9 K 2023-04-20 12:55:58,954 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-20 12:55:58,955 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for bed4cc4c061ff4314d49ea7add6e3672 in 1256ms, sequenceid=424, compaction requested=true 2023-04-20 12:55:58,955 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:55:58,955 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:58,955 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:55:58,955 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:55:58,955 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:58,955 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:55:58,955 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:55:58,956 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:58,956 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:58,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:58,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:58,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:58,956 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:58,956 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:55:58,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:58,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:58,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:58,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:58,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:58,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:55:58,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a83ee63[Count = 0] remaining members to acquire global barrier 2023-04-20 12:55:58,957 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:55:58,957 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:58,958 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:58,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:58,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:58,958 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:55:58,958 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:55:58,958 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:55:58,958 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:58,958 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:55:58,959 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:58,959 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:55:58,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:58,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:58,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:58,959 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:58,959 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:55:58,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:58,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:58,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:58,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:58,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:58,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:58,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:58,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:55:58,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:55:58,960 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:55:58,960 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:55:58,960 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:55:58,960 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:55:58,960 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:55:58,961 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:58,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:58,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:55:58,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:55:58,961 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:58,961 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:58,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:58,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:58,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:58,962 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:58,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:55:58,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:58,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:55:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:58,963 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:55:58,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:55:58,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:58,964 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:58,964 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:58,964 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:58,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:58,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:58,964 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:55:58,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:55:58,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:55:58,964 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:55:58,965 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:55:58,964 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:58,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:55:58,965 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:55:58,965 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:58,965 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:55:58,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:55:58,965 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:58,965 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:55:58,965 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:59,065 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:55:59,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:55:59,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:55:59,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:55:59,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:55:59,067 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:55:59,067 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:59,067 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:55:59,067 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:55:59,068 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:59,068 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:59,068 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:55:59,068 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:59,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:55:59,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:55:59,068 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:59,068 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:55:59,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:59,069 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:55:59,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:55:59,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:55:59,069 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:55:59,069 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:55:59,069 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:55:59,069 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:55:59,069 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:55:59,069 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:55:59,069 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:55:59,069 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:55:59,069 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:55:59,070 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:55:59,070 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing bed4cc4c061ff4314d49ea7add6e3672 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-20 12:55:59,070 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=A 2023-04-20 12:55:59,070 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:59,070 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=B 2023-04-20 12:55:59,070 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:59,070 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=C 2023-04-20 12:55:59,070 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:55:59,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on bed4cc4c061ff4314d49ea7add6e3672 2023-04-20 12:55:59,184 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:55:59,204 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:59,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47054 deadline: 1681995419198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:59,204 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:59,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54262 deadline: 1681995419202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:59,205 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:59,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54244 deadline: 1681995419203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:59,308 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:59,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47054 deadline: 1681995419305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:59,309 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:59,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54262 deadline: 1681995419305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:59,309 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:59,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54244 deadline: 1681995419305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:59,478 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=442 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/785e6d157b5a4a0fbef58def6b4c2fcb 2023-04-20 12:55:59,512 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:59,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54262 deadline: 1681995419510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:59,513 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:59,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47054 deadline: 1681995419510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:59,513 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:59,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54244 deadline: 1681995419511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:59,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:59,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54244 deadline: 1681995419814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:59,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:59,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54262 deadline: 1681995419815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:59,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:55:59,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47054 deadline: 1681995419815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:55:59,885 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=442 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/26d96d4775424aabbc9c6e99b94486f7 2023-04-20 12:56:00,293 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=442 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/b38c3ff404e14d2fb44e451e9c52283d 2023-04-20 12:56:00,296 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/785e6d157b5a4a0fbef58def6b4c2fcb as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/785e6d157b5a4a0fbef58def6b4c2fcb 2023-04-20 12:56:00,299 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/785e6d157b5a4a0fbef58def6b4c2fcb, entries=150, sequenceid=442, filesize=11.9 K 2023-04-20 12:56:00,299 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/26d96d4775424aabbc9c6e99b94486f7 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/26d96d4775424aabbc9c6e99b94486f7 2023-04-20 12:56:00,302 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/26d96d4775424aabbc9c6e99b94486f7, entries=150, sequenceid=442, filesize=11.9 K 2023-04-20 12:56:00,302 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/b38c3ff404e14d2fb44e451e9c52283d as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/b38c3ff404e14d2fb44e451e9c52283d 2023-04-20 12:56:00,305 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/b38c3ff404e14d2fb44e451e9c52283d, entries=150, sequenceid=442, filesize=11.9 K 2023-04-20 12:56:00,306 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for bed4cc4c061ff4314d49ea7add6e3672 in 1236ms, sequenceid=442, compaction requested=true 2023-04-20 12:56:00,306 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:56:00,306 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:56:00,306 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:00,306 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:00,306 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,306 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:00,306 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:00,307 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,307 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:00,307 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:00,307 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:00,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:00,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:00,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:00,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:00,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:00,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:00,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:00,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c268e48[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:00,309 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:00,309 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:00,309 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:00,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:00,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:00,309 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:00,310 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:00,310 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:00,310 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,310 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:00,310 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:00,310 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,310 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:00,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:00,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:00,310 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:00,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:00,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:00,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:00,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:00,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:00,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:00,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:00,312 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:00,312 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:00,312 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:00,312 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:00,312 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:00,313 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:00,313 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:00,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:00,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:00,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:00,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:00,313 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:00,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:00,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:00,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:00,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:00,314 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:00,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:00,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:00,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:00,315 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:00,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:00,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,317 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,317 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:00,317 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:00,317 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:00,317 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,317 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:00,317 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:00,317 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:00,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:00,317 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:00,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:00,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:00,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:00,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:00,317 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:00,317 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:00,317 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:00,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:00,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:00,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on bed4cc4c061ff4314d49ea7add6e3672 2023-04-20 12:56:00,324 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bed4cc4c061ff4314d49ea7add6e3672 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-20 12:56:00,325 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=A 2023-04-20 12:56:00,325 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:00,325 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=B 2023-04-20 12:56:00,325 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:00,325 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=C 2023-04-20 12:56:00,325 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:00,330 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=458 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/0c3bf915ae11430e9ea756c692353fa7 2023-04-20 12:56:00,360 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:00,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54244 deadline: 1681995420353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,361 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:00,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47054 deadline: 1681995420354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,365 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:00,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54262 deadline: 1681995420355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,418 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:00,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:00,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:00,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:00,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:00,420 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:00,420 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:00,420 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:00,420 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:00,421 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:00,421 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:00,421 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:00,421 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:00,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:00,422 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,422 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:00,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:00,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:00,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:00,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:00,422 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:00,422 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:00,422 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:00,423 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:00,423 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:00,423 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:00,423 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:00,423 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:56:00,423 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:00,423 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:56:00,423 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:56:00,423 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:56:00,423 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:00,423 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:00,423 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,423 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:00,423 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:00,424 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,424 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:00,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:00,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:00,424 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:00,424 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:00,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:00,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:00,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:00,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:00,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:00,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15b1195a[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:00,425 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:00,425 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:00,426 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:00,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:00,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:00,426 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:00,426 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,426 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:00,426 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:00,426 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:00,427 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,427 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:00,427 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:00,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:00,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:00,427 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:00,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:00,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:00,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:00,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:00,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:00,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:00,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:00,428 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:00,428 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:00,428 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:00,428 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:00,428 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:00,429 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:00,429 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:00,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:00,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:00,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:00,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:00,429 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:00,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:00,430 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:00,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:00,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:00,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:00,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:00,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:00,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:00,430 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:00,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:00,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,432 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,432 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:00,432 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:00,432 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,432 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:00,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:00,432 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:00,432 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:00,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:00,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:00,432 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:00,432 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:00,432 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:00,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:00,432 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:00,432 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:00,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:00,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:00,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:00,464 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:00,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54244 deadline: 1681995420461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,464 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:00,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47054 deadline: 1681995420462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,468 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:00,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54262 deadline: 1681995420466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,532 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:00,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:00,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:00,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:00,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:00,535 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:00,535 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:00,535 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:00,535 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:00,535 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:00,535 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:00,536 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:00,536 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:00,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:00,536 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,536 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:00,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:00,537 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:00,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:00,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:00,537 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:00,537 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:00,537 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:00,537 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:00,537 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:00,537 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:00,537 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:00,537 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:00,537 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:56:00,537 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:56:00,538 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:56:00,538 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:56:00,538 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:00,538 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:00,538 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,538 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:00,538 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:00,539 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:00,539 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:00,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:00,539 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:00,539 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:00,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:00,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:00,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:00,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:00,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:00,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@118b1647[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:00,541 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:00,541 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:00,541 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:00,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:00,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:00,541 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:00,541 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:00,541 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:00,541 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,541 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:00,542 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,542 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:00,542 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:00,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:00,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:00,542 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:00,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:00,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:00,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:00,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:00,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:00,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:00,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:00,544 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:00,544 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:00,544 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:00,544 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:00,544 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:00,545 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:00,545 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:00,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:00,545 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:00,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:00,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:00,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:00,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:00,545 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:00,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:00,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:00,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:00,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:00,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:00,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:00,546 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:00,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:00,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,548 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,548 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:00,548 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:00,548 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:00,548 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:00,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:00,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:00,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:00,548 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:00,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:00,548 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:00,548 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:00,549 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:00,549 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:00,549 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:00,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:00,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:00,549 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:00,649 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:00,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:00,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:00,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:00,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:00,651 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:00,651 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:00,652 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:00,652 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:00,652 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:00,652 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:00,652 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:00,652 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:00,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:00,653 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:00,653 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:00,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:00,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:00,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:00,653 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:00,654 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:00,654 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:00,654 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:00,654 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:00,654 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:00,654 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:00,654 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:56:00,654 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:56:00,654 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:00,654 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:56:00,655 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:56:00,655 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:00,655 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:00,655 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,655 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:00,655 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:00,656 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,656 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:00,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:00,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:00,656 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:00,656 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:00,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:00,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:00,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:00,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:00,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:00,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@244e02e3[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:00,658 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:00,658 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:00,658 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:00,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:00,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:00,658 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:00,658 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:00,658 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:00,658 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,659 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:00,659 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:00,659 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:00,659 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,659 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:00,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:00,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:00,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:00,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:00,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:00,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:00,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:00,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:00,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:00,661 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:00,661 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:00,661 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:00,661 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:00,661 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:00,662 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:00,662 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:00,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:00,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:00,662 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:00,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:00,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:00,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:00,662 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:00,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:00,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:00,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:00,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:00,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:00,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:00,663 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:00,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:00,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,665 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,665 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:00,665 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:00,665 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,665 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:00,665 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:00,665 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:00,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:00,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:00,665 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:00,665 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:00,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:00,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:00,665 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:00,666 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:00,666 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:00,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:00,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:00,666 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:00,671 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:00,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54244 deadline: 1681995420666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,671 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:00,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47054 deadline: 1681995420666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,675 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:00,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54262 deadline: 1681995420670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,739 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=458 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/a53fbd26d96341408b94fcfc9dbb04a0 2023-04-20 12:56:00,766 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:00,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:00,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:00,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:00,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:00,769 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:00,769 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:00,769 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:00,769 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:00,769 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:00,769 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:00,770 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:00,771 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:00,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:00,771 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,771 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:00,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:00,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:00,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:00,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:00,771 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:00,772 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:00,772 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:00,772 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:00,772 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:00,772 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:00,772 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:00,772 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:00,772 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:56:00,772 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. started... 2023-04-20 12:56:00,772 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. as already flushing 2023-04-20 12:56:00,772 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:56:00,772 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:00,772 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:00,772 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,772 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:00,772 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:00,773 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,773 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:00,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:00,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:00,773 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:00,773 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:00,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:00,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:00,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:00,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:00,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:00,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c628cec[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:00,774 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:00,774 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:00,775 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:00,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:00,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:00,775 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:00,775 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:00,775 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:00,775 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,775 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:00,776 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:00,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:00,776 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:00,776 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:00,776 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:00,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:00,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:00,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:00,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:00,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:00,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:00,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:00,777 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:00,777 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:00,777 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:00,777 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:00,777 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:00,778 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:00,778 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:00,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:00,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:00,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:00,778 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:00,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:00,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:00,779 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:00,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:00,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:00,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:00,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:00,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:00,779 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:00,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:00,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:00,781 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,781 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:00,781 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:00,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:00,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:00,781 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:00,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:00,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:00,781 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:00,781 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:00,781 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,781 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:00,781 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:00,781 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:00,781 DEBUG [Thread-3089] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:00,781 DEBUG [Thread-3089] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:00,781 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:00,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:00,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:00,824 DEBUG [Thread-3092] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6acb8caa to 127.0.0.1:56673 2023-04-20 12:56:00,824 DEBUG [Thread-3092] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:56:00,829 DEBUG [Thread-3096] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x25d2db45 to 127.0.0.1:56673 2023-04-20 12:56:00,829 DEBUG [Thread-3096] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:56:00,831 DEBUG [Thread-3098] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c2ce5cd to 127.0.0.1:56673 2023-04-20 12:56:00,831 DEBUG [Thread-3098] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:56:00,833 DEBUG [Thread-3090] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1ee0ad26 to 127.0.0.1:56673 2023-04-20 12:56:00,833 DEBUG [Thread-3090] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:56:00,835 DEBUG [Thread-3094] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x70263db3 to 127.0.0.1:56673 2023-04-20 12:56:00,835 DEBUG [Thread-3094] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:56:00,881 DEBUG [Thread-3089] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:00,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:00,974 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:00,974 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:00,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54244 deadline: 1681995420974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:47054 deadline: 1681995420974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:00,977 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:00,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:54262 deadline: 1681995420977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:01,146 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=458 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/e849de5039b4463e8f4dc168216ff8b4 2023-04-20 12:56:01,152 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/0c3bf915ae11430e9ea756c692353fa7 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/0c3bf915ae11430e9ea756c692353fa7 2023-04-20 12:56:01,158 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/0c3bf915ae11430e9ea756c692353fa7, entries=250, sequenceid=458, filesize=16.7 K 2023-04-20 12:56:01,160 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/a53fbd26d96341408b94fcfc9dbb04a0 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/a53fbd26d96341408b94fcfc9dbb04a0 2023-04-20 12:56:01,166 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/a53fbd26d96341408b94fcfc9dbb04a0, entries=150, sequenceid=458, filesize=11.9 K 2023-04-20 12:56:01,167 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/e849de5039b4463e8f4dc168216ff8b4 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/e849de5039b4463e8f4dc168216ff8b4 2023-04-20 12:56:01,173 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/e849de5039b4463e8f4dc168216ff8b4, entries=150, sequenceid=458, filesize=11.9 K 2023-04-20 12:56:01,174 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for bed4cc4c061ff4314d49ea7add6e3672 in 850ms, sequenceid=458, compaction requested=true 2023-04-20 12:56:01,174 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:56:01,174 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:56:01,174 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:56:01,174 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-20 12:56:01,174 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-20 12:56:01,174 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:56:01,176 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 66696 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-20 12:56:01,176 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61816 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-20 12:56:01,176 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): bed4cc4c061ff4314d49ea7add6e3672/A is initiating minor compaction (all files) 2023-04-20 12:56:01,176 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): bed4cc4c061ff4314d49ea7add6e3672/B is initiating minor compaction (all files) 2023-04-20 12:56:01,176 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bed4cc4c061ff4314d49ea7add6e3672/A in TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:56:01,176 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bed4cc4c061ff4314d49ea7add6e3672/B in TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:56:01,176 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/4ff21af4bfae4c4785eb6f97b8603295, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/bc73d51254314ad4935c7e74720d796a, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/01f03213ceea4a0388285672079cb527, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/785e6d157b5a4a0fbef58def6b4c2fcb, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/0c3bf915ae11430e9ea756c692353fa7] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp, totalSize=65.1 K 2023-04-20 12:56:01,176 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/a35cd88c88514174b48d7c26998ad1de, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/cc440fccb4034f9eba49efe9a0738fc8, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/5405bf40d349446dad18422f9ddf04f2, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/26d96d4775424aabbc9c6e99b94486f7, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/a53fbd26d96341408b94fcfc9dbb04a0] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp, totalSize=60.4 K 2023-04-20 12:56:01,177 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 4ff21af4bfae4c4785eb6f97b8603295, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=389, earliestPutTs=1681995355116 2023-04-20 12:56:01,177 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting a35cd88c88514174b48d7c26998ad1de, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=389, earliestPutTs=1681995355116 2023-04-20 12:56:01,177 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting bc73d51254314ad4935c7e74720d796a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1681995355766 2023-04-20 12:56:01,177 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting cc440fccb4034f9eba49efe9a0738fc8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1681995355766 2023-04-20 12:56:01,177 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 01f03213ceea4a0388285672079cb527, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=424, earliestPutTs=1681995356907 2023-04-20 12:56:01,178 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 5405bf40d349446dad18422f9ddf04f2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=424, earliestPutTs=1681995356907 2023-04-20 12:56:01,178 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 785e6d157b5a4a0fbef58def6b4c2fcb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=442, earliestPutTs=1681995358054 2023-04-20 12:56:01,178 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 26d96d4775424aabbc9c6e99b94486f7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=442, earliestPutTs=1681995358054 2023-04-20 12:56:01,178 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 0c3bf915ae11430e9ea756c692353fa7, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=458, earliestPutTs=1681995359196 2023-04-20 12:56:01,178 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting a53fbd26d96341408b94fcfc9dbb04a0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=458, earliestPutTs=1681995359196 2023-04-20 12:56:01,191 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): bed4cc4c061ff4314d49ea7add6e3672#A#compaction#472 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:56:01,192 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): bed4cc4c061ff4314d49ea7add6e3672#B#compaction#473 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:56:01,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on bed4cc4c061ff4314d49ea7add6e3672 2023-04-20 12:56:01,287 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing bed4cc4c061ff4314d49ea7add6e3672 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-20 12:56:01,287 DEBUG [Thread-3079] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4048dcb4 to 127.0.0.1:56673 2023-04-20 12:56:01,287 DEBUG [Thread-3079] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:56:01,287 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=A 2023-04-20 12:56:01,287 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:01,287 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=B 2023-04-20 12:56:01,288 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:01,288 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=C 2023-04-20 12:56:01,288 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:01,379 DEBUG [Thread-3087] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c9c1feb to 127.0.0.1:56673 2023-04-20 12:56:01,379 DEBUG [Thread-3087] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:56:01,482 DEBUG [Thread-3081] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4e7a5e9a to 127.0.0.1:56673 2023-04-20 12:56:01,482 DEBUG [Thread-3081] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:56:01,483 DEBUG [Thread-3083] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x62447900 to 127.0.0.1:56673 2023-04-20 12:56:01,483 DEBUG [Thread-3083] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:56:01,485 DEBUG [Thread-3085] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x519db5f2 to 127.0.0.1:56673 2023-04-20 12:56:01,485 DEBUG [Thread-3085] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:56:01,485 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-20 12:56:01,485 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(394): wrote 28 2023-04-20 12:56:01,485 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(394): wrote 114 2023-04-20 12:56:01,486 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(394): wrote 108 2023-04-20 12:56:01,486 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(394): wrote 109 2023-04-20 12:56:01,486 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(394): wrote 23 2023-04-20 12:56:01,486 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-20 12:56:01,486 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-20 12:56:01,486 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(402): scanned 2122 2023-04-20 12:56:01,486 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(403): verified 6366 rows 2023-04-20 12:56:01,486 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(402): scanned 2111 2023-04-20 12:56:01,486 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(403): verified 6333 rows 2023-04-20 12:56:01,486 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(402): scanned 2114 2023-04-20 12:56:01,487 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(403): verified 6342 rows 2023-04-20 12:56:01,487 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(402): scanned 2116 2023-04-20 12:56:01,487 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(403): verified 6348 rows 2023-04-20 12:56:01,487 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(402): scanned 2117 2023-04-20 12:56:01,487 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(403): verified 6351 rows 2023-04-20 12:56:01,487 INFO [Listener at localhost.localdomain/42205] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-20 12:56:01,487 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0817e2e5 to 127.0.0.1:56673 2023-04-20 12:56:01,487 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:56:01,489 INFO [Listener at localhost.localdomain/42205] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-20 12:56:01,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-20 12:56:01,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-20 12:56:01,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-20 12:56:01,493 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681995361493"}]},"ts":"1681995361493"} 2023-04-20 12:56:01,494 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-20 12:56:01,502 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-20 12:56:01,503 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=bed4cc4c061ff4314d49ea7add6e3672, UNASSIGN}] 2023-04-20 12:56:01,504 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=bed4cc4c061ff4314d49ea7add6e3672, UNASSIGN 2023-04-20 12:56:01,505 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=bed4cc4c061ff4314d49ea7add6e3672, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:01,505 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681995361504"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681995361504"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681995361504"}]},"ts":"1681995361504"} 2023-04-20 12:56:01,506 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947}] 2023-04-20 12:56:01,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-20 12:56:01,605 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/0ef9135258574ae9a9c0fc322e967c4c as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/0ef9135258574ae9a9c0fc322e967c4c 2023-04-20 12:56:01,608 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/6c1d479f8fcd47a79bc60514839982f5 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/6c1d479f8fcd47a79bc60514839982f5 2023-04-20 12:56:01,611 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in bed4cc4c061ff4314d49ea7add6e3672/B of bed4cc4c061ff4314d49ea7add6e3672 into 0ef9135258574ae9a9c0fc322e967c4c(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:56:01,611 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:56:01,611 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672., storeName=bed4cc4c061ff4314d49ea7add6e3672/B, priority=11, startTime=1681995361174; duration=0sec 2023-04-20 12:56:01,611 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:56:01,611 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-20 12:56:01,613 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61816 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-20 12:56:01,613 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): bed4cc4c061ff4314d49ea7add6e3672/C is initiating minor compaction (all files) 2023-04-20 12:56:01,613 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bed4cc4c061ff4314d49ea7add6e3672/C in TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:56:01,613 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/f4cf31c96e714e979e444a023bd61206, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/3b40cd5f4bd547499063bd4723b019a8, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/747db884724b4ae986da46a6dc588d9c, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/b38c3ff404e14d2fb44e451e9c52283d, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/e849de5039b4463e8f4dc168216ff8b4] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp, totalSize=60.4 K 2023-04-20 12:56:01,613 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in bed4cc4c061ff4314d49ea7add6e3672/A of bed4cc4c061ff4314d49ea7add6e3672 into 6c1d479f8fcd47a79bc60514839982f5(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:56:01,613 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:56:01,613 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672., storeName=bed4cc4c061ff4314d49ea7add6e3672/A, priority=11, startTime=1681995361174; duration=0sec 2023-04-20 12:56:01,613 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting f4cf31c96e714e979e444a023bd61206, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=389, earliestPutTs=1681995355116 2023-04-20 12:56:01,613 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:56:01,614 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 3b40cd5f4bd547499063bd4723b019a8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1681995355766 2023-04-20 12:56:01,614 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 747db884724b4ae986da46a6dc588d9c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=424, earliestPutTs=1681995356907 2023-04-20 12:56:01,614 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting b38c3ff404e14d2fb44e451e9c52283d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=442, earliestPutTs=1681995358054 2023-04-20 12:56:01,615 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting e849de5039b4463e8f4dc168216ff8b4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=458, earliestPutTs=1681995359196 2023-04-20 12:56:01,622 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): bed4cc4c061ff4314d49ea7add6e3672#C#compaction#475 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:56:01,658 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close bed4cc4c061ff4314d49ea7add6e3672 2023-04-20 12:56:01,658 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing bed4cc4c061ff4314d49ea7add6e3672, disabling compactions & flushes 2023-04-20 12:56:01,658 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1858): waiting for 1 compactions & cache flush to complete for region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:56:01,700 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=475 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/346037794bbc4ec280790cb1b7dec16b 2023-04-20 12:56:01,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-20 12:56:02,029 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/3e27131db3c54224ab6f63283b554e16 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/3e27131db3c54224ab6f63283b554e16 2023-04-20 12:56:02,033 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in bed4cc4c061ff4314d49ea7add6e3672/C of bed4cc4c061ff4314d49ea7add6e3672 into 3e27131db3c54224ab6f63283b554e16(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:56:02,033 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:56:02,033 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:56:02,033 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672., storeName=bed4cc4c061ff4314d49ea7add6e3672/C, priority=11, startTime=1681995361174; duration=0sec 2023-04-20 12:56:02,033 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:56:02,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-20 12:56:02,119 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=475 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/2022ce40a67942d9be1512a8b2fc1b82 2023-04-20 12:56:02,537 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=475 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/b904daf27a2c4e57aca0171e99197a8a 2023-04-20 12:56:02,548 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/346037794bbc4ec280790cb1b7dec16b as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/346037794bbc4ec280790cb1b7dec16b 2023-04-20 12:56:02,552 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/346037794bbc4ec280790cb1b7dec16b, entries=150, sequenceid=475, filesize=11.9 K 2023-04-20 12:56:02,553 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/2022ce40a67942d9be1512a8b2fc1b82 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/2022ce40a67942d9be1512a8b2fc1b82 2023-04-20 12:56:02,558 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/2022ce40a67942d9be1512a8b2fc1b82, entries=150, sequenceid=475, filesize=11.9 K 2023-04-20 12:56:02,559 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/b904daf27a2c4e57aca0171e99197a8a as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/b904daf27a2c4e57aca0171e99197a8a 2023-04-20 12:56:02,566 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/b904daf27a2c4e57aca0171e99197a8a, entries=150, sequenceid=475, filesize=11.9 K 2023-04-20 12:56:02,567 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=26.84 KB/27480 for bed4cc4c061ff4314d49ea7add6e3672 in 1280ms, sequenceid=475, compaction requested=false 2023-04-20 12:56:02,567 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:56:02,567 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:56:02,567 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:56:02,567 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. after waiting 0 ms 2023-04-20 12:56:02,567 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:56:02,567 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing bed4cc4c061ff4314d49ea7add6e3672 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-20 12:56:02,567 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=A 2023-04-20 12:56:02,568 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:02,568 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=B 2023-04-20 12:56:02,568 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:02,568 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK bed4cc4c061ff4314d49ea7add6e3672, store=C 2023-04-20 12:56:02,568 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:02,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-20 12:56:02,978 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=485 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/4caab8d1fe7748be9c98aa9c017edeb0 2023-04-20 12:56:02,993 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=485 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/5784f7f8cea8429f9b0bbde6fb11466c 2023-04-20 12:56:03,409 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=485 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/90824f39303f4e648bd542fde0b03478 2023-04-20 12:56:03,420 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/A/4caab8d1fe7748be9c98aa9c017edeb0 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/4caab8d1fe7748be9c98aa9c017edeb0 2023-04-20 12:56:03,426 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/4caab8d1fe7748be9c98aa9c017edeb0, entries=150, sequenceid=485, filesize=11.9 K 2023-04-20 12:56:03,427 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/B/5784f7f8cea8429f9b0bbde6fb11466c as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/5784f7f8cea8429f9b0bbde6fb11466c 2023-04-20 12:56:03,433 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/5784f7f8cea8429f9b0bbde6fb11466c, entries=150, sequenceid=485, filesize=11.9 K 2023-04-20 12:56:03,434 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/.tmp/C/90824f39303f4e648bd542fde0b03478 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/90824f39303f4e648bd542fde0b03478 2023-04-20 12:56:03,439 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/90824f39303f4e648bd542fde0b03478, entries=150, sequenceid=485, filesize=11.9 K 2023-04-20 12:56:03,440 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for bed4cc4c061ff4314d49ea7add6e3672 in 873ms, sequenceid=485, compaction requested=true 2023-04-20 12:56:03,463 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/eccfdf12947f4871a6967f33fdbd6513, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/fa56b98987e74bc6957d962f0890b1ce, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/a4c07c0e94314eb8b28dfc257636743d, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/5634f84ae66941349a7597c5d60d3f86, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/7cddb6ea1ec348e9a4861af368551fee, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/cfaf857cfd9f4d1484cad268026e6991, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/be45eb10900f48ac9dafcd62954a767b, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/4f83e69989314dbcb8a877d17d453191, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/cc265e856a6c4f4a86619cb3c976e3e5, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/4caa68eee05b480598992e38f1b18389, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/d39746c2ae814a0d83b069e7abcdae2d, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/b66fabb8f0f440a393e596520960776e, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/2e410e3c741342df9498f0977c17702d, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/f541889e4a96431283662bb1fa8d7857, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/38e4ad0867124600b853234eb09d7173, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/3c9b511f7299498893955aad9f578ec0, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/36514fed58694c39a437fe4699331219, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/b8b462643aaa488fa22686eb00de3e72, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/633cc9c9743c4d2fb064b7f3e4eaf52b, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/c952d98f537e4d9aa11dd71394d87f50, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/e6a756def45a438c8751f0461bb2612d, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/9b4b7ac06f9346bea6e3a44142aefb6b, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/e751512dda3141aea4db4d9279ac5c43, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/88c6ab2656d94fb18d0b30581d9bd45f, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/7a5062d3dc9a4d98abe411d30b45ded3, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/ce641c767154433aa4e3780f3d8212d4, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/4ff21af4bfae4c4785eb6f97b8603295, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/bc73d51254314ad4935c7e74720d796a, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/01f03213ceea4a0388285672079cb527, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/785e6d157b5a4a0fbef58def6b4c2fcb, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/0c3bf915ae11430e9ea756c692353fa7] to archive 2023-04-20 12:56:03,464 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-20 12:56:03,465 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/eccfdf12947f4871a6967f33fdbd6513 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/eccfdf12947f4871a6967f33fdbd6513 2023-04-20 12:56:03,466 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/fa56b98987e74bc6957d962f0890b1ce to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/fa56b98987e74bc6957d962f0890b1ce 2023-04-20 12:56:03,467 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/a4c07c0e94314eb8b28dfc257636743d to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/a4c07c0e94314eb8b28dfc257636743d 2023-04-20 12:56:03,467 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/5634f84ae66941349a7597c5d60d3f86 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/5634f84ae66941349a7597c5d60d3f86 2023-04-20 12:56:03,468 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/7cddb6ea1ec348e9a4861af368551fee to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/7cddb6ea1ec348e9a4861af368551fee 2023-04-20 12:56:03,469 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/cfaf857cfd9f4d1484cad268026e6991 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/cfaf857cfd9f4d1484cad268026e6991 2023-04-20 12:56:03,470 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/be45eb10900f48ac9dafcd62954a767b to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/be45eb10900f48ac9dafcd62954a767b 2023-04-20 12:56:03,471 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/4f83e69989314dbcb8a877d17d453191 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/4f83e69989314dbcb8a877d17d453191 2023-04-20 12:56:03,471 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/cc265e856a6c4f4a86619cb3c976e3e5 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/cc265e856a6c4f4a86619cb3c976e3e5 2023-04-20 12:56:03,472 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/4caa68eee05b480598992e38f1b18389 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/4caa68eee05b480598992e38f1b18389 2023-04-20 12:56:03,473 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/d39746c2ae814a0d83b069e7abcdae2d to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/d39746c2ae814a0d83b069e7abcdae2d 2023-04-20 12:56:03,473 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/b66fabb8f0f440a393e596520960776e to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/b66fabb8f0f440a393e596520960776e 2023-04-20 12:56:03,474 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/2e410e3c741342df9498f0977c17702d to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/2e410e3c741342df9498f0977c17702d 2023-04-20 12:56:03,476 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/f541889e4a96431283662bb1fa8d7857 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/f541889e4a96431283662bb1fa8d7857 2023-04-20 12:56:03,477 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/38e4ad0867124600b853234eb09d7173 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/38e4ad0867124600b853234eb09d7173 2023-04-20 12:56:03,478 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/3c9b511f7299498893955aad9f578ec0 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/3c9b511f7299498893955aad9f578ec0 2023-04-20 12:56:03,479 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/36514fed58694c39a437fe4699331219 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/36514fed58694c39a437fe4699331219 2023-04-20 12:56:03,480 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/b8b462643aaa488fa22686eb00de3e72 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/b8b462643aaa488fa22686eb00de3e72 2023-04-20 12:56:03,481 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/633cc9c9743c4d2fb064b7f3e4eaf52b to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/633cc9c9743c4d2fb064b7f3e4eaf52b 2023-04-20 12:56:03,481 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/c952d98f537e4d9aa11dd71394d87f50 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/c952d98f537e4d9aa11dd71394d87f50 2023-04-20 12:56:03,482 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/e6a756def45a438c8751f0461bb2612d to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/e6a756def45a438c8751f0461bb2612d 2023-04-20 12:56:03,483 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/9b4b7ac06f9346bea6e3a44142aefb6b to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/9b4b7ac06f9346bea6e3a44142aefb6b 2023-04-20 12:56:03,485 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/e751512dda3141aea4db4d9279ac5c43 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/e751512dda3141aea4db4d9279ac5c43 2023-04-20 12:56:03,486 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/88c6ab2656d94fb18d0b30581d9bd45f to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/88c6ab2656d94fb18d0b30581d9bd45f 2023-04-20 12:56:03,487 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/7a5062d3dc9a4d98abe411d30b45ded3 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/7a5062d3dc9a4d98abe411d30b45ded3 2023-04-20 12:56:03,488 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/ce641c767154433aa4e3780f3d8212d4 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/ce641c767154433aa4e3780f3d8212d4 2023-04-20 12:56:03,489 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/4ff21af4bfae4c4785eb6f97b8603295 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/4ff21af4bfae4c4785eb6f97b8603295 2023-04-20 12:56:03,490 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/bc73d51254314ad4935c7e74720d796a to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/bc73d51254314ad4935c7e74720d796a 2023-04-20 12:56:03,491 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/01f03213ceea4a0388285672079cb527 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/01f03213ceea4a0388285672079cb527 2023-04-20 12:56:03,492 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/785e6d157b5a4a0fbef58def6b4c2fcb to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/785e6d157b5a4a0fbef58def6b4c2fcb 2023-04-20 12:56:03,494 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/0c3bf915ae11430e9ea756c692353fa7 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/0c3bf915ae11430e9ea756c692353fa7 2023-04-20 12:56:03,498 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/9d68a7fcb1864ef3bdbc52c9b82459ec, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/7c0222ce61b343d9b7849e5c37eff46a, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/cc36db3460ca49d6a4976d6a790f8b5e, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/739606c1b6d9493786fad675e2d11c1e, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/1a61932f7cc041dc8b1053bf00818529, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/43a0cfa0def049c392188bdd28df2005, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/18ea54d6fc2e4aa3a28a99e7083cd327, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/40a86b744558418da08bb3a6bec972c4, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/bdc728ee7c904c85a926570806640adf, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/4d21231fed0a4e63b06b46e442c0cbb2, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/6358f051bdca4ca5baa2d86a1cbc594f, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/dd95a4d8a98e4d5ab36b3df2f44f4d34, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/662f2d05bf45400289316be46c6d67e2, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/b81b0d91142443f0a47cff02f1f11b4b, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/26aaf31cf51441d79f5faffbe3f1ffcf, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/3d060eea3def4f318ee361534c53e147, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/e4b248211dcf414ba2352fa5095aded5, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/7bf9bf4cd0cd4983a354c2c1021f2ef6, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/1f7bd7186f4f453e9a203b651a3e4dc8, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/5bf62e40ef604331ad398d58ea0c954a, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/5d2c401500a141ee83820384ad2be706, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/bad6ee59968948f5bc17410fd689c35b, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/8575608848b341f398ce7b5adea8980a, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/8104f4660b5f418cb131c5ba752086a1, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/f9810e2b4588428389d78bbf0e1dcc8b, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/a35cd88c88514174b48d7c26998ad1de, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/5f745d7eae6241a58fb42adb39daa4f2, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/cc440fccb4034f9eba49efe9a0738fc8, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/5405bf40d349446dad18422f9ddf04f2, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/26d96d4775424aabbc9c6e99b94486f7, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/a53fbd26d96341408b94fcfc9dbb04a0] to archive 2023-04-20 12:56:03,499 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-20 12:56:03,501 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/9d68a7fcb1864ef3bdbc52c9b82459ec to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/9d68a7fcb1864ef3bdbc52c9b82459ec 2023-04-20 12:56:03,502 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/7c0222ce61b343d9b7849e5c37eff46a to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/7c0222ce61b343d9b7849e5c37eff46a 2023-04-20 12:56:03,503 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/cc36db3460ca49d6a4976d6a790f8b5e to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/cc36db3460ca49d6a4976d6a790f8b5e 2023-04-20 12:56:03,504 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/739606c1b6d9493786fad675e2d11c1e to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/739606c1b6d9493786fad675e2d11c1e 2023-04-20 12:56:03,505 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/1a61932f7cc041dc8b1053bf00818529 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/1a61932f7cc041dc8b1053bf00818529 2023-04-20 12:56:03,506 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/43a0cfa0def049c392188bdd28df2005 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/43a0cfa0def049c392188bdd28df2005 2023-04-20 12:56:03,507 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/18ea54d6fc2e4aa3a28a99e7083cd327 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/18ea54d6fc2e4aa3a28a99e7083cd327 2023-04-20 12:56:03,508 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/40a86b744558418da08bb3a6bec972c4 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/40a86b744558418da08bb3a6bec972c4 2023-04-20 12:56:03,509 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/bdc728ee7c904c85a926570806640adf to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/bdc728ee7c904c85a926570806640adf 2023-04-20 12:56:03,510 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/4d21231fed0a4e63b06b46e442c0cbb2 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/4d21231fed0a4e63b06b46e442c0cbb2 2023-04-20 12:56:03,511 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/6358f051bdca4ca5baa2d86a1cbc594f to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/6358f051bdca4ca5baa2d86a1cbc594f 2023-04-20 12:56:03,511 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/dd95a4d8a98e4d5ab36b3df2f44f4d34 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/dd95a4d8a98e4d5ab36b3df2f44f4d34 2023-04-20 12:56:03,512 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/662f2d05bf45400289316be46c6d67e2 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/662f2d05bf45400289316be46c6d67e2 2023-04-20 12:56:03,513 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/b81b0d91142443f0a47cff02f1f11b4b to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/b81b0d91142443f0a47cff02f1f11b4b 2023-04-20 12:56:03,514 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/26aaf31cf51441d79f5faffbe3f1ffcf to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/26aaf31cf51441d79f5faffbe3f1ffcf 2023-04-20 12:56:03,515 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/3d060eea3def4f318ee361534c53e147 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/3d060eea3def4f318ee361534c53e147 2023-04-20 12:56:03,516 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/e4b248211dcf414ba2352fa5095aded5 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/e4b248211dcf414ba2352fa5095aded5 2023-04-20 12:56:03,517 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/7bf9bf4cd0cd4983a354c2c1021f2ef6 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/7bf9bf4cd0cd4983a354c2c1021f2ef6 2023-04-20 12:56:03,518 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/1f7bd7186f4f453e9a203b651a3e4dc8 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/1f7bd7186f4f453e9a203b651a3e4dc8 2023-04-20 12:56:03,519 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/5bf62e40ef604331ad398d58ea0c954a to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/5bf62e40ef604331ad398d58ea0c954a 2023-04-20 12:56:03,520 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/5d2c401500a141ee83820384ad2be706 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/5d2c401500a141ee83820384ad2be706 2023-04-20 12:56:03,521 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/bad6ee59968948f5bc17410fd689c35b to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/bad6ee59968948f5bc17410fd689c35b 2023-04-20 12:56:03,522 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/8575608848b341f398ce7b5adea8980a to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/8575608848b341f398ce7b5adea8980a 2023-04-20 12:56:03,523 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/8104f4660b5f418cb131c5ba752086a1 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/8104f4660b5f418cb131c5ba752086a1 2023-04-20 12:56:03,524 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/f9810e2b4588428389d78bbf0e1dcc8b to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/f9810e2b4588428389d78bbf0e1dcc8b 2023-04-20 12:56:03,525 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/a35cd88c88514174b48d7c26998ad1de to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/a35cd88c88514174b48d7c26998ad1de 2023-04-20 12:56:03,526 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/5f745d7eae6241a58fb42adb39daa4f2 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/5f745d7eae6241a58fb42adb39daa4f2 2023-04-20 12:56:03,527 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/cc440fccb4034f9eba49efe9a0738fc8 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/cc440fccb4034f9eba49efe9a0738fc8 2023-04-20 12:56:03,528 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/5405bf40d349446dad18422f9ddf04f2 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/5405bf40d349446dad18422f9ddf04f2 2023-04-20 12:56:03,529 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/26d96d4775424aabbc9c6e99b94486f7 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/26d96d4775424aabbc9c6e99b94486f7 2023-04-20 12:56:03,529 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/a53fbd26d96341408b94fcfc9dbb04a0 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/a53fbd26d96341408b94fcfc9dbb04a0 2023-04-20 12:56:03,533 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/24a31044ddda451fbfe6116973c2411d, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/25d46b1616954b22ab16e3eb9cb8dd4f, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/a4e407d11b2e440a9c2679cab00ec5a0, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/98bb90f666dc43dbbbf4615088928fb7, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/d598f73a75ee4296abcea320c44dd3c0, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/7d67ced317a9483580ac5c683cfc9ab5, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/f45f6f12d5b049b1b6caf833ee52a3ab, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/98de14f7e898410aa6aaceaa998bfc6d, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/0ed6702a5dc94f42a8c81f7926d8f219, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/b87fc7f34f124cbc9138305a3b36c026, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/890060bf910542a8ad8f67e0e0c170b6, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/a899124b43be4e738b62bc4dac5b7a19, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/a90d582e84094defbc91492388725ebf, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/e68164dd50ce481fac6ce5cb4ae53c09, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/edb995b8299c4ceb94e3d758e4eab0f1, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/ba43d43ba99d4f16831f97be463ccee0, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/6ea823ef6a854ad4afb010a2e7ae12a9, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/4c486af44e7645578cb9a672a5cd6880, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/8b9a4a865144454b964ffc775dd2889e, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/561cb9b4e85b4c819ef6e65b206adb5e, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/12e870813b964fab963bd7b327c4361d, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/b10dab1e4b5d48178a1dcd76f2330e2e, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/2459d714e30940d5a71b84b0c22b57f6, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/9c9b67e893a2454d86e7a80aae0104eb, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/8eb784835dae4672b0c8c7a943ee1e32, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/f4cf31c96e714e979e444a023bd61206, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/90f1c8f37327410e9f06a588e4820955, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/3b40cd5f4bd547499063bd4723b019a8, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/747db884724b4ae986da46a6dc588d9c, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/b38c3ff404e14d2fb44e451e9c52283d, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/e849de5039b4463e8f4dc168216ff8b4] to archive 2023-04-20 12:56:03,534 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-20 12:56:03,535 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/24a31044ddda451fbfe6116973c2411d to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/24a31044ddda451fbfe6116973c2411d 2023-04-20 12:56:03,536 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/25d46b1616954b22ab16e3eb9cb8dd4f to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/25d46b1616954b22ab16e3eb9cb8dd4f 2023-04-20 12:56:03,537 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/a4e407d11b2e440a9c2679cab00ec5a0 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/a4e407d11b2e440a9c2679cab00ec5a0 2023-04-20 12:56:03,538 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/98bb90f666dc43dbbbf4615088928fb7 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/98bb90f666dc43dbbbf4615088928fb7 2023-04-20 12:56:03,539 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/d598f73a75ee4296abcea320c44dd3c0 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/d598f73a75ee4296abcea320c44dd3c0 2023-04-20 12:56:03,540 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/7d67ced317a9483580ac5c683cfc9ab5 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/7d67ced317a9483580ac5c683cfc9ab5 2023-04-20 12:56:03,541 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/f45f6f12d5b049b1b6caf833ee52a3ab to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/f45f6f12d5b049b1b6caf833ee52a3ab 2023-04-20 12:56:03,541 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/98de14f7e898410aa6aaceaa998bfc6d to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/98de14f7e898410aa6aaceaa998bfc6d 2023-04-20 12:56:03,542 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/0ed6702a5dc94f42a8c81f7926d8f219 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/0ed6702a5dc94f42a8c81f7926d8f219 2023-04-20 12:56:03,543 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/b87fc7f34f124cbc9138305a3b36c026 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/b87fc7f34f124cbc9138305a3b36c026 2023-04-20 12:56:03,544 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/890060bf910542a8ad8f67e0e0c170b6 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/890060bf910542a8ad8f67e0e0c170b6 2023-04-20 12:56:03,545 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/a899124b43be4e738b62bc4dac5b7a19 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/a899124b43be4e738b62bc4dac5b7a19 2023-04-20 12:56:03,546 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/a90d582e84094defbc91492388725ebf to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/a90d582e84094defbc91492388725ebf 2023-04-20 12:56:03,546 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/e68164dd50ce481fac6ce5cb4ae53c09 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/e68164dd50ce481fac6ce5cb4ae53c09 2023-04-20 12:56:03,547 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/edb995b8299c4ceb94e3d758e4eab0f1 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/edb995b8299c4ceb94e3d758e4eab0f1 2023-04-20 12:56:03,548 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/ba43d43ba99d4f16831f97be463ccee0 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/ba43d43ba99d4f16831f97be463ccee0 2023-04-20 12:56:03,549 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/6ea823ef6a854ad4afb010a2e7ae12a9 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/6ea823ef6a854ad4afb010a2e7ae12a9 2023-04-20 12:56:03,549 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/4c486af44e7645578cb9a672a5cd6880 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/4c486af44e7645578cb9a672a5cd6880 2023-04-20 12:56:03,550 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/8b9a4a865144454b964ffc775dd2889e to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/8b9a4a865144454b964ffc775dd2889e 2023-04-20 12:56:03,550 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/561cb9b4e85b4c819ef6e65b206adb5e to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/561cb9b4e85b4c819ef6e65b206adb5e 2023-04-20 12:56:03,551 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/12e870813b964fab963bd7b327c4361d to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/12e870813b964fab963bd7b327c4361d 2023-04-20 12:56:03,552 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/b10dab1e4b5d48178a1dcd76f2330e2e to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/b10dab1e4b5d48178a1dcd76f2330e2e 2023-04-20 12:56:03,552 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/2459d714e30940d5a71b84b0c22b57f6 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/2459d714e30940d5a71b84b0c22b57f6 2023-04-20 12:56:03,553 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/9c9b67e893a2454d86e7a80aae0104eb to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/9c9b67e893a2454d86e7a80aae0104eb 2023-04-20 12:56:03,553 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/8eb784835dae4672b0c8c7a943ee1e32 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/8eb784835dae4672b0c8c7a943ee1e32 2023-04-20 12:56:03,554 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/f4cf31c96e714e979e444a023bd61206 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/f4cf31c96e714e979e444a023bd61206 2023-04-20 12:56:03,555 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/90f1c8f37327410e9f06a588e4820955 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/90f1c8f37327410e9f06a588e4820955 2023-04-20 12:56:03,556 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/3b40cd5f4bd547499063bd4723b019a8 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/3b40cd5f4bd547499063bd4723b019a8 2023-04-20 12:56:03,556 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/747db884724b4ae986da46a6dc588d9c to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/747db884724b4ae986da46a6dc588d9c 2023-04-20 12:56:03,557 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/b38c3ff404e14d2fb44e451e9c52283d to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/b38c3ff404e14d2fb44e451e9c52283d 2023-04-20 12:56:03,558 DEBUG [StoreCloser-TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/e849de5039b4463e8f4dc168216ff8b4 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/e849de5039b4463e8f4dc168216ff8b4 2023-04-20 12:56:03,560 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/recovered.edits/488.seqid, newMaxSeqId=488, maxSeqId=1 2023-04-20 12:56:03,561 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672. 2023-04-20 12:56:03,562 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for bed4cc4c061ff4314d49ea7add6e3672: 2023-04-20 12:56:03,563 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed bed4cc4c061ff4314d49ea7add6e3672 2023-04-20 12:56:03,564 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=bed4cc4c061ff4314d49ea7add6e3672, regionState=CLOSED 2023-04-20 12:56:03,564 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681995363564"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681995363564"}]},"ts":"1681995363564"} 2023-04-20 12:56:03,566 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-04-20 12:56:03,566 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure bed4cc4c061ff4314d49ea7add6e3672, server=jenkins-hbase20.apache.org,36277,1681995213947 in 2.0590 sec 2023-04-20 12:56:03,568 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-04-20 12:56:03,568 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=bed4cc4c061ff4314d49ea7add6e3672, UNASSIGN in 2.0630 sec 2023-04-20 12:56:03,568 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681995363568"}]},"ts":"1681995363568"} 2023-04-20 12:56:03,569 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-20 12:56:03,570 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-20 12:56:03,572 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.0820 sec 2023-04-20 12:56:03,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-20 12:56:03,600 INFO [Listener at localhost.localdomain/42205] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-04-20 12:56:03,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-20 12:56:03,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 12:56:03,603 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 12:56:03,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-20 12:56:03,604 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 12:56:03,610 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672 2023-04-20 12:56:03,614 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A, FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B, FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C, FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/recovered.edits] 2023-04-20 12:56:03,620 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/346037794bbc4ec280790cb1b7dec16b to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/346037794bbc4ec280790cb1b7dec16b 2023-04-20 12:56:03,623 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/4caab8d1fe7748be9c98aa9c017edeb0 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/4caab8d1fe7748be9c98aa9c017edeb0 2023-04-20 12:56:03,626 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/6c1d479f8fcd47a79bc60514839982f5 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/A/6c1d479f8fcd47a79bc60514839982f5 2023-04-20 12:56:03,630 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/0ef9135258574ae9a9c0fc322e967c4c to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/0ef9135258574ae9a9c0fc322e967c4c 2023-04-20 12:56:03,633 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/2022ce40a67942d9be1512a8b2fc1b82 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/2022ce40a67942d9be1512a8b2fc1b82 2023-04-20 12:56:03,635 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/5784f7f8cea8429f9b0bbde6fb11466c to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/B/5784f7f8cea8429f9b0bbde6fb11466c 2023-04-20 12:56:03,640 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/3e27131db3c54224ab6f63283b554e16 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/3e27131db3c54224ab6f63283b554e16 2023-04-20 12:56:03,642 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/90824f39303f4e648bd542fde0b03478 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/90824f39303f4e648bd542fde0b03478 2023-04-20 12:56:03,644 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/b904daf27a2c4e57aca0171e99197a8a to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/C/b904daf27a2c4e57aca0171e99197a8a 2023-04-20 12:56:03,648 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/recovered.edits/488.seqid to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672/recovered.edits/488.seqid 2023-04-20 12:56:03,649 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/bed4cc4c061ff4314d49ea7add6e3672 2023-04-20 12:56:03,649 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-20 12:56:03,652 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 12:56:03,654 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-20 12:56:03,657 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-20 12:56:03,658 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 12:56:03,658 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-20 12:56:03,659 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681995363658"}]},"ts":"9223372036854775807"} 2023-04-20 12:56:03,661 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-20 12:56:03,661 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => bed4cc4c061ff4314d49ea7add6e3672, NAME => 'TestAcidGuarantees,,1681995338627.bed4cc4c061ff4314d49ea7add6e3672.', STARTKEY => '', ENDKEY => ''}] 2023-04-20 12:56:03,661 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-20 12:56:03,662 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681995363661"}]},"ts":"9223372036854775807"} 2023-04-20 12:56:03,664 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-20 12:56:03,666 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 12:56:03,667 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 65 msec 2023-04-20 12:56:03,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-20 12:56:03,706 INFO [Listener at localhost.localdomain/42205] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-04-20 12:56:03,721 INFO [Listener at localhost.localdomain/42205] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=232 (was 235), OpenFileDescriptor=552 (was 537) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=346 (was 305) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 167), AvailableMemoryMB=9448 (was 9010) - AvailableMemoryMB LEAK? - 2023-04-20 12:56:03,731 INFO [Listener at localhost.localdomain/42205] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=232, OpenFileDescriptor=552, MaxFileDescriptor=60000, SystemLoadAverage=346, ProcessCount=167, AvailableMemoryMB=9448 2023-04-20 12:56:03,733 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-20 12:56:03,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-20 12:56:03,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-20 12:56:03,735 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-20 12:56:03,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-04-20 12:56:03,736 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-20 12:56:03,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-20 12:56:03,737 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8 2023-04-20 12:56:03,738 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8 empty. 2023-04-20 12:56:03,738 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8 2023-04-20 12:56:03,738 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-20 12:56:03,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-20 12:56:04,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-20 12:56:04,145 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-20 12:56:04,146 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 8113dc3e39e3420c86d60a8170c364f8, NAME => 'TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp 2023-04-20 12:56:04,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-20 12:56:04,553 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-20 12:56:04,553 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 8113dc3e39e3420c86d60a8170c364f8, disabling compactions & flushes 2023-04-20 12:56:04,553 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:04,553 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:04,553 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. after waiting 0 ms 2023-04-20 12:56:04,553 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:04,554 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:04,554 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 8113dc3e39e3420c86d60a8170c364f8: 2023-04-20 12:56:04,558 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-20 12:56:04,561 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681995364561"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681995364561"}]},"ts":"1681995364561"} 2023-04-20 12:56:04,563 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-20 12:56:04,564 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-20 12:56:04,564 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681995364564"}]},"ts":"1681995364564"} 2023-04-20 12:56:04,565 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-20 12:56:04,568 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8113dc3e39e3420c86d60a8170c364f8, ASSIGN}] 2023-04-20 12:56:04,570 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8113dc3e39e3420c86d60a8170c364f8, ASSIGN 2023-04-20 12:56:04,571 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=8113dc3e39e3420c86d60a8170c364f8, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,36277,1681995213947; forceNewPlan=false, retain=false 2023-04-20 12:56:04,723 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=8113dc3e39e3420c86d60a8170c364f8, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:04,723 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681995364723"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681995364723"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681995364723"}]},"ts":"1681995364723"} 2023-04-20 12:56:04,726 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure 8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947}] 2023-04-20 12:56:04,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-20 12:56:04,886 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:04,887 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 8113dc3e39e3420c86d60a8170c364f8, NAME => 'TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.', STARTKEY => '', ENDKEY => ''} 2023-04-20 12:56:04,887 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 8113dc3e39e3420c86d60a8170c364f8 2023-04-20 12:56:04,887 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-20 12:56:04,887 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 8113dc3e39e3420c86d60a8170c364f8 2023-04-20 12:56:04,888 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 8113dc3e39e3420c86d60a8170c364f8 2023-04-20 12:56:04,890 INFO [StoreOpener-8113dc3e39e3420c86d60a8170c364f8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 8113dc3e39e3420c86d60a8170c364f8 2023-04-20 12:56:04,892 DEBUG [StoreOpener-8113dc3e39e3420c86d60a8170c364f8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A 2023-04-20 12:56:04,892 DEBUG [StoreOpener-8113dc3e39e3420c86d60a8170c364f8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A 2023-04-20 12:56:04,892 INFO [StoreOpener-8113dc3e39e3420c86d60a8170c364f8-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 12:56:04,892 INFO [StoreOpener-8113dc3e39e3420c86d60a8170c364f8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8113dc3e39e3420c86d60a8170c364f8 columnFamilyName A 2023-04-20 12:56:04,893 INFO [StoreOpener-8113dc3e39e3420c86d60a8170c364f8-1] regionserver.HStore(310): Store=8113dc3e39e3420c86d60a8170c364f8/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 12:56:04,893 INFO [StoreOpener-8113dc3e39e3420c86d60a8170c364f8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 8113dc3e39e3420c86d60a8170c364f8 2023-04-20 12:56:04,894 DEBUG [StoreOpener-8113dc3e39e3420c86d60a8170c364f8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B 2023-04-20 12:56:04,895 DEBUG [StoreOpener-8113dc3e39e3420c86d60a8170c364f8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B 2023-04-20 12:56:04,895 INFO [StoreOpener-8113dc3e39e3420c86d60a8170c364f8-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 12:56:04,895 INFO [StoreOpener-8113dc3e39e3420c86d60a8170c364f8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8113dc3e39e3420c86d60a8170c364f8 columnFamilyName B 2023-04-20 12:56:04,896 INFO [StoreOpener-8113dc3e39e3420c86d60a8170c364f8-1] regionserver.HStore(310): Store=8113dc3e39e3420c86d60a8170c364f8/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 12:56:04,896 INFO [StoreOpener-8113dc3e39e3420c86d60a8170c364f8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 8113dc3e39e3420c86d60a8170c364f8 2023-04-20 12:56:04,897 DEBUG [StoreOpener-8113dc3e39e3420c86d60a8170c364f8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C 2023-04-20 12:56:04,897 DEBUG [StoreOpener-8113dc3e39e3420c86d60a8170c364f8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C 2023-04-20 12:56:04,898 INFO [StoreOpener-8113dc3e39e3420c86d60a8170c364f8-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 12:56:04,898 INFO [StoreOpener-8113dc3e39e3420c86d60a8170c364f8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8113dc3e39e3420c86d60a8170c364f8 columnFamilyName C 2023-04-20 12:56:04,898 INFO [StoreOpener-8113dc3e39e3420c86d60a8170c364f8-1] regionserver.HStore(310): Store=8113dc3e39e3420c86d60a8170c364f8/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 12:56:04,898 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:04,899 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8 2023-04-20 12:56:04,900 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8 2023-04-20 12:56:04,902 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-20 12:56:04,904 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 8113dc3e39e3420c86d60a8170c364f8 2023-04-20 12:56:04,906 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-20 12:56:04,906 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 8113dc3e39e3420c86d60a8170c364f8; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69671459, jitterRate=0.038185641169548035}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-20 12:56:04,906 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 8113dc3e39e3420c86d60a8170c364f8: 2023-04-20 12:56:04,907 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8., pid=56, masterSystemTime=1681995364878 2023-04-20 12:56:04,908 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:04,909 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:04,909 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=8113dc3e39e3420c86d60a8170c364f8, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:04,909 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681995364909"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1681995364909"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681995364909"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681995364909"}]},"ts":"1681995364909"} 2023-04-20 12:56:04,912 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-04-20 12:56:04,912 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure 8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 in 185 msec 2023-04-20 12:56:04,914 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-04-20 12:56:04,914 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=8113dc3e39e3420c86d60a8170c364f8, ASSIGN in 344 msec 2023-04-20 12:56:04,915 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-20 12:56:04,915 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681995364915"}]},"ts":"1681995364915"} 2023-04-20 12:56:04,916 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-20 12:56:04,918 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-20 12:56:04,920 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1850 sec 2023-04-20 12:56:05,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-20 12:56:05,845 INFO [Listener at localhost.localdomain/42205] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-04-20 12:56:05,846 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x1bb3d2c9 to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:56:05,855 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@27acfde4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:56:05,857 DEBUG [Listener at localhost.localdomain/42205] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:56:05,859 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41966, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:56:05,861 DEBUG [Listener at localhost.localdomain/42205] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-20 12:56:05,865 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:33860, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-20 12:56:05,868 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-20 12:56:05,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.HMaster$12(2454): Client=jenkins//148.251.75.209 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-20 12:56:05,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-20 12:56:06,279 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-20 12:56:06,280 DEBUG [PEWorker-2] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-20 12:56:06,280 INFO [PEWorker-2] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-20 12:56:06,283 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-20 12:56:06,286 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8113dc3e39e3420c86d60a8170c364f8, REOPEN/MOVE}] 2023-04-20 12:56:06,287 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8113dc3e39e3420c86d60a8170c364f8, REOPEN/MOVE 2023-04-20 12:56:06,287 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=8113dc3e39e3420c86d60a8170c364f8, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:06,287 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681995366287"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681995366287"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681995366287"}]},"ts":"1681995366287"} 2023-04-20 12:56:06,288 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure 8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947}] 2023-04-20 12:56:06,440 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 8113dc3e39e3420c86d60a8170c364f8 2023-04-20 12:56:06,441 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 8113dc3e39e3420c86d60a8170c364f8, disabling compactions & flushes 2023-04-20 12:56:06,441 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:06,442 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:06,442 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. after waiting 0 ms 2023-04-20 12:56:06,442 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:06,445 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-20 12:56:06,445 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:06,445 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 8113dc3e39e3420c86d60a8170c364f8: 2023-04-20 12:56:06,445 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 8113dc3e39e3420c86d60a8170c364f8 to self. 2023-04-20 12:56:06,447 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 8113dc3e39e3420c86d60a8170c364f8 2023-04-20 12:56:06,447 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=8113dc3e39e3420c86d60a8170c364f8, regionState=CLOSED 2023-04-20 12:56:06,447 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681995366447"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681995366447"}]},"ts":"1681995366447"} 2023-04-20 12:56:06,449 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-04-20 12:56:06,449 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure 8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 in 160 msec 2023-04-20 12:56:06,449 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=8113dc3e39e3420c86d60a8170c364f8, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase20.apache.org,36277,1681995213947; forceNewPlan=false, retain=true 2023-04-20 12:56:06,600 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=8113dc3e39e3420c86d60a8170c364f8, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:06,600 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681995366600"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681995366600"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681995366600"}]},"ts":"1681995366600"} 2023-04-20 12:56:06,602 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure 8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947}] 2023-04-20 12:56:06,764 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:06,764 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 8113dc3e39e3420c86d60a8170c364f8, NAME => 'TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.', STARTKEY => '', ENDKEY => ''} 2023-04-20 12:56:06,765 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 8113dc3e39e3420c86d60a8170c364f8 2023-04-20 12:56:06,765 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-20 12:56:06,765 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 8113dc3e39e3420c86d60a8170c364f8 2023-04-20 12:56:06,765 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 8113dc3e39e3420c86d60a8170c364f8 2023-04-20 12:56:06,768 INFO [StoreOpener-8113dc3e39e3420c86d60a8170c364f8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 8113dc3e39e3420c86d60a8170c364f8 2023-04-20 12:56:06,769 DEBUG [StoreOpener-8113dc3e39e3420c86d60a8170c364f8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A 2023-04-20 12:56:06,769 DEBUG [StoreOpener-8113dc3e39e3420c86d60a8170c364f8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A 2023-04-20 12:56:06,769 INFO [StoreOpener-8113dc3e39e3420c86d60a8170c364f8-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 12:56:06,769 INFO [StoreOpener-8113dc3e39e3420c86d60a8170c364f8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8113dc3e39e3420c86d60a8170c364f8 columnFamilyName A 2023-04-20 12:56:06,771 INFO [StoreOpener-8113dc3e39e3420c86d60a8170c364f8-1] regionserver.HStore(310): Store=8113dc3e39e3420c86d60a8170c364f8/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 12:56:06,771 INFO [StoreOpener-8113dc3e39e3420c86d60a8170c364f8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 8113dc3e39e3420c86d60a8170c364f8 2023-04-20 12:56:06,772 DEBUG [StoreOpener-8113dc3e39e3420c86d60a8170c364f8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B 2023-04-20 12:56:06,772 DEBUG [StoreOpener-8113dc3e39e3420c86d60a8170c364f8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B 2023-04-20 12:56:06,772 INFO [StoreOpener-8113dc3e39e3420c86d60a8170c364f8-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 12:56:06,773 INFO [StoreOpener-8113dc3e39e3420c86d60a8170c364f8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8113dc3e39e3420c86d60a8170c364f8 columnFamilyName B 2023-04-20 12:56:06,773 INFO [StoreOpener-8113dc3e39e3420c86d60a8170c364f8-1] regionserver.HStore(310): Store=8113dc3e39e3420c86d60a8170c364f8/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 12:56:06,773 INFO [StoreOpener-8113dc3e39e3420c86d60a8170c364f8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 8113dc3e39e3420c86d60a8170c364f8 2023-04-20 12:56:06,774 DEBUG [StoreOpener-8113dc3e39e3420c86d60a8170c364f8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C 2023-04-20 12:56:06,774 DEBUG [StoreOpener-8113dc3e39e3420c86d60a8170c364f8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C 2023-04-20 12:56:06,774 INFO [StoreOpener-8113dc3e39e3420c86d60a8170c364f8-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-20 12:56:06,774 INFO [StoreOpener-8113dc3e39e3420c86d60a8170c364f8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8113dc3e39e3420c86d60a8170c364f8 columnFamilyName C 2023-04-20 12:56:06,775 INFO [StoreOpener-8113dc3e39e3420c86d60a8170c364f8-1] regionserver.HStore(310): Store=8113dc3e39e3420c86d60a8170c364f8/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-20 12:56:06,775 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:06,776 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8 2023-04-20 12:56:06,777 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8 2023-04-20 12:56:06,779 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-20 12:56:06,780 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 8113dc3e39e3420c86d60a8170c364f8 2023-04-20 12:56:06,781 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 8113dc3e39e3420c86d60a8170c364f8; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65060381, jitterRate=-0.03052477538585663}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-20 12:56:06,781 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 8113dc3e39e3420c86d60a8170c364f8: 2023-04-20 12:56:06,781 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8., pid=61, masterSystemTime=1681995366755 2023-04-20 12:56:06,783 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:06,783 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:06,783 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=8113dc3e39e3420c86d60a8170c364f8, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:06,784 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681995366783"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1681995366783"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681995366783"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681995366783"}]},"ts":"1681995366783"} 2023-04-20 12:56:06,786 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-04-20 12:56:06,786 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure 8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 in 183 msec 2023-04-20 12:56:06,788 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-04-20 12:56:06,788 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=8113dc3e39e3420c86d60a8170c364f8, REOPEN/MOVE in 500 msec 2023-04-20 12:56:06,790 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-04-20 12:56:06,790 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 506 msec 2023-04-20 12:56:06,792 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 923 msec 2023-04-20 12:56:06,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-04-20 12:56:06,795 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x055c027c to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:56:06,799 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@fcbd84, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:56:06,800 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x7d22e3b3 to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:56:06,804 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@18884189, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:56:06,804 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x44a601bf to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:56:06,809 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2ef34de3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:56:06,809 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x67bf0d7e to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:56:06,814 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@136043cf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:56:06,814 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x3d52316c to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:56:06,819 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@22990f45, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:56:06,820 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x7c686287 to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:56:06,825 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@16e1a968, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:56:06,825 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x28c646e5 to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:56:06,830 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@17856520, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:56:06,830 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x1069406b to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:56:06,840 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@a82cded, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:56:06,840 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x1ba9b9c7 to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:56:06,848 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4119c8c1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:56:06,849 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(139): Connect 0x7bf3fb2a to 127.0.0.1:56673 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-20 12:56:06,853 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2dc21212, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-20 12:56:06,864 DEBUG [hconnection-0x14b4c47a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:56:06,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:06,866 DEBUG [hconnection-0x2a88b49c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:56:06,866 DEBUG [hconnection-0x6aaf4df7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:56:06,867 DEBUG [hconnection-0x731201e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:56:06,867 DEBUG [hconnection-0x4e267027-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:56:06,867 DEBUG [hconnection-0x55fcc0cb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:56:06,867 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41978, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:56:06,870 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41992, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:56:06,870 DEBUG [hconnection-0x65b47b4b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:56:06,872 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42008, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:56:06,870 DEBUG [hconnection-0x3e2b1e49-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:56:06,870 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41990, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:56:06,873 DEBUG [hconnection-0x13091900-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:56:06,873 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42042, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:56:06,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:06,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:06,875 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:06,875 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:06,875 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:06,875 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:06,875 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:06,875 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:06,877 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:06,877 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:06,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:06,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:06,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:06,877 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42060, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:56:06,878 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:06,878 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:06,878 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:06,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:06,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:06,878 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:06,878 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:06,878 DEBUG [hconnection-0x3a22463f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-20 12:56:06,879 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42016, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:56:06,878 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:06,879 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:06,879 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:06,879 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:06,879 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:06,879 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:06,879 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:06,880 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:06,884 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8113dc3e39e3420c86d60a8170c364f8: 2023-04-20 12:56:06,885 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:06,885 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:06,885 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:06,885 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42018, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:56:06,885 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:06,885 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:06,885 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:06,885 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42056, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:56:06,885 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42076, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-20 12:56:06,889 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:06,890 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:06,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:06,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:06,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:06,890 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:06,890 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:06,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:06,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:06,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 8113dc3e39e3420c86d60a8170c364f8 2023-04-20 12:56:06,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:06,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:06,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:06,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:06,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@410425f3[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:06,895 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:06,895 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:06,895 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8113dc3e39e3420c86d60a8170c364f8 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-20 12:56:06,896 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=A 2023-04-20 12:56:06,896 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:06,896 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=B 2023-04-20 12:56:06,896 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:06,896 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=C 2023-04-20 12:56:06,896 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:06,902 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:06,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:06,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:06,902 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:06,902 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:06,902 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:06,902 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:06,903 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:06,906 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:06,906 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:06,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42008 deadline: 1681995426904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:06,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:06,906 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:06,906 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:06,906 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:06,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:06,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:06,906 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:06,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42016 deadline: 1681995426905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:06,907 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:06,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42018 deadline: 1681995426905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:06,908 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:06,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42076 deadline: 1681995426907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:06,908 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:06,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41978 deadline: 1681995426907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:06,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:06,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:06,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:06,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:06,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:06,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:06,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:06,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:06,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:06,916 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:06,916 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:06,916 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:06,916 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:06,916 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:06,919 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:06,919 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:06,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:06,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:06,919 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:06,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:06,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:06,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:06,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:06,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:06,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:06,920 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:06,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:06,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:06,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:06,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:06,921 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:06,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:06,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:06,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:06,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:06,922 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:06,922 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:06,922 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:06,922 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:06,922 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:06,922 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:06,922 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:06,922 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:06,922 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:06,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:06,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:06,922 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:06,923 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:06,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:06,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:06,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:06,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:06,923 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:06,923 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:07,008 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:07,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42008 deadline: 1681995427007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,008 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:07,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42016 deadline: 1681995427007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,010 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:07,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42076 deadline: 1681995427008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,010 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:07,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42018 deadline: 1681995427009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,010 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:07,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41978 deadline: 1681995427009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,023 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:07,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:07,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:07,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:07,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:07,026 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:07,026 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:07,026 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:07,026 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:07,026 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,026 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:07,027 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:07,027 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:07,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:07,027 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,027 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:07,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:07,027 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:07,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:07,027 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:07,028 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:07,028 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:07,028 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:07,028 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:07,028 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:07,028 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:07,028 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:07,028 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:07,028 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:07,028 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:07,028 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:07,028 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:07,028 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:07,028 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,028 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:07,028 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:07,029 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,029 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:07,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:07,029 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,029 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:07,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:07,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:07,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:07,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:07,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8f42aba[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:07,030 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:07,030 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,031 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,031 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:07,031 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:07,031 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:07,031 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,031 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:07,032 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,032 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:07,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:07,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:07,032 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:07,032 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:07,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:07,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:07,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:07,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:07,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:07,034 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:07,034 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:07,034 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:07,034 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:07,034 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:07,038 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,038 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:07,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:07,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,038 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:07,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:07,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:07,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:07,038 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:07,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,039 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:07,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,040 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:07,040 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:07,040 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,040 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:07,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:07,040 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:07,040 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:07,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:07,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:07,040 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:07,040 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,041 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,041 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,041 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,041 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:07,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:07,041 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:07,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:07,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:07,141 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:07,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:07,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:07,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:07,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:07,144 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:07,144 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:07,144 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:07,144 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:07,144 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,144 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:07,145 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:07,145 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:07,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:07,145 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,145 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:07,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:07,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:07,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:07,145 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:07,146 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:07,146 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:07,146 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:07,146 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:07,146 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:07,146 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:07,146 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:07,146 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:07,146 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:07,146 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:07,146 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:07,146 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:07,146 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:07,146 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,146 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:07,146 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:07,147 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:07,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:07,147 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:07,147 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,147 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:07,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:07,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:07,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:07,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c79685d[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:07,148 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:07,148 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,149 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,149 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:07,149 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:07,149 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:07,149 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,149 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:07,150 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:07,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:07,150 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:07,150 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:07,150 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:07,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:07,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:07,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:07,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:07,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:07,152 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:07,152 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:07,152 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:07,152 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:07,152 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:07,153 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,153 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,153 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:07,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:07,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:07,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,153 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:07,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:07,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:07,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:07,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,155 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:07,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,156 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:07,156 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:07,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:07,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:07,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:07,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:07,157 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,157 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:07,157 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:07,157 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:07,157 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:07,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:07,157 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,157 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:07,157 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,157 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,157 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:07,157 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:07,212 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:07,212 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:07,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42008 deadline: 1681995427210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42016 deadline: 1681995427210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,212 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:07,212 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:07,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41978 deadline: 1681995427211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42018 deadline: 1681995427211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,212 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:07,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42076 deadline: 1681995427211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,257 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:07,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:07,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:07,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:07,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:07,260 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:07,260 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:07,260 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:07,260 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:07,260 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,260 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:07,261 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,261 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:07,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:07,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:07,261 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,261 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:07,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:07,262 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:07,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:07,262 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:07,262 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:07,262 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:07,262 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:07,262 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:07,262 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:07,262 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:07,262 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:07,262 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:07,262 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:07,262 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:07,262 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:07,262 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:07,262 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:07,263 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,263 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:07,263 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:07,263 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,263 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:07,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:07,264 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,264 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:07,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:07,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:07,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:07,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:07,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14b49ad2[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:07,264 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:07,265 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,265 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,265 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:07,265 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:07,265 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:07,265 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,265 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:07,266 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,266 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:07,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:07,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:07,266 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:07,266 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:07,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:07,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:07,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:07,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:07,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:07,267 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:07,267 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:07,267 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:07,267 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:07,268 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:07,268 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,268 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:07,268 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:07,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:07,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:07,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:07,269 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:07,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:07,270 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:07,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,272 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:07,272 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:07,272 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:07,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:07,272 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:07,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:07,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:07,272 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:07,272 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:07,272 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,272 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,272 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:07,272 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:07,272 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:07,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:07,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:07,272 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,327 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420ff11f43851ee43629f37ec89f4fd4cfc to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420ff11f43851ee43629f37ec89f4fd4cfc 2023-04-20 12:56:07,372 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:07,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:07,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:07,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:07,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:07,376 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:07,376 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:07,376 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:07,376 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:07,376 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,376 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:07,377 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,377 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:07,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:07,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:07,377 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,377 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:07,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:07,377 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:07,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:07,377 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:07,378 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:07,378 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:07,378 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:07,378 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:07,378 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:07,378 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:07,378 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:07,378 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:07,378 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:07,378 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:07,378 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:07,378 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:07,378 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:07,378 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,378 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:07,378 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:07,379 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,379 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:07,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:07,379 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,379 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:07,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:07,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:07,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:07,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:07,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@747f0e0a[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:07,381 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:07,381 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,381 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,381 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:07,381 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:07,381 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:07,381 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,381 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:07,382 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:07,382 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:07,382 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,382 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:07,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:07,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:07,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:07,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:07,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:07,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:07,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:07,384 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:07,384 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:07,384 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:07,384 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:07,384 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:07,385 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,385 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,385 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:07,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:07,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:07,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:07,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:07,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:07,386 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:07,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,390 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:07,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,402 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,402 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:07,402 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:07,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:07,402 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:07,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:07,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:07,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:07,402 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:07,402 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:07,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:07,403 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:07,403 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:07,403 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:07,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:07,403 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,403 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,403 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,403 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,503 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:07,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:07,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:07,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:07,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:07,505 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:07,505 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:07,505 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:07,505 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:07,506 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,506 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:07,506 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,506 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:07,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:07,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:07,506 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,506 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:07,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:07,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:07,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:07,507 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:07,507 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:07,507 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:07,507 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:07,507 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:07,507 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:07,508 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:07,508 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:07,508 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:07,508 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:07,508 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:07,508 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:07,508 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:07,508 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:07,508 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,508 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:07,508 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:07,508 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:07,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:07,509 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:07,509 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,509 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:07,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:07,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:07,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:07,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d774bc5[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:07,510 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:07,510 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,510 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,510 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,510 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:07,510 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:07,510 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:07,511 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:07,511 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,511 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:07,511 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:07,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:07,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:07,511 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:07,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:07,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:07,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:07,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:07,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:07,513 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:07,513 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:07,513 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:07,513 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:07,513 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:07,514 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:07,514 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42018 deadline: 1681995427513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:07,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:07,514 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,514 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:07,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,514 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:07,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:07,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:07,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,515 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:07,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42008 deadline: 1681995427513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,515 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:07,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42076 deadline: 1681995427514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:07,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,516 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:07,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,516 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:07,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,516 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:07,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42016 deadline: 1681995427514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41978 deadline: 1681995427515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,517 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,517 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:07,517 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:07,517 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:07,517 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:07,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:07,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:07,517 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:07,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:07,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:07,517 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:07,517 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:07,517 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:07,517 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:07,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:07,517 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,518 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,518 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,618 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:07,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:07,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:07,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:07,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:07,624 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:07,624 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:07,624 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:07,624 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:07,624 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,624 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:07,625 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,625 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:07,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:07,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:07,625 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,625 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:07,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:07,625 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:07,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:07,625 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:07,625 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:07,626 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:07,626 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:07,626 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:07,626 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:07,626 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:07,626 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:07,626 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:07,626 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:07,626 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:07,626 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:07,626 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:07,626 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,626 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:07,626 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:07,627 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,627 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:07,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:07,627 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,627 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:07,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:07,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:07,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:07,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:07,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a5bfbb6[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:07,628 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:07,628 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,628 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,629 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:07,629 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:07,629 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,629 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:07,629 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:07,629 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,629 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:07,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:07,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:07,629 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:07,629 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:07,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:07,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:07,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:07,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:07,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:07,631 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:07,631 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:07,631 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:07,631 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:07,631 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:07,632 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,632 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:07,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:07,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,632 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:07,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:07,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:07,632 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:07,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:07,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,633 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:07,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,634 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,634 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:07,634 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:07,634 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,634 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,634 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:07,634 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:07,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:07,634 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:07,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:07,635 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:07,634 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:07,635 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,635 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:07,635 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:07,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:07,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:07,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:07,731 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=22, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/A/8db65877342f457990d6b47df3d2be63 2023-04-20 12:56:07,735 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:07,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:07,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:07,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:07,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:07,738 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:07,738 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:07,738 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:07,738 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:07,739 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,739 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:07,740 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,740 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:07,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:07,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:07,741 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:07,741 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:07,741 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:07,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:07,741 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:07,741 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:07,742 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:07,742 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:07,742 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:07,742 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:07,742 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:07,742 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:07,742 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:07,742 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:07,742 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:07,742 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:07,742 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:07,742 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:07,742 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,742 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:07,742 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:07,743 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,743 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:07,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:07,743 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,743 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:07,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:07,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:07,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:07,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:07,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a899a25[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:07,744 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:07,744 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,744 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,745 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:07,745 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:07,745 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:07,745 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,745 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:07,745 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,745 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:07,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:07,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:07,745 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:07,745 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:07,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:07,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:07,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:07,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:07,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:07,747 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:07,747 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:07,747 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:07,747 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:07,747 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:07,748 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,748 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:07,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:07,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:07,749 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:07,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,749 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,748 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:07,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:07,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:07,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:07,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,753 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:07,753 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,753 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:07,753 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:07,753 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,753 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,753 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,753 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,753 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:07,753 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:07,753 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:07,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:07,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:07,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:07,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:07,754 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:07,754 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:07,754 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:07,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:07,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:07,854 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:07,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:07,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:07,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:07,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:07,856 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:07,857 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:07,857 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:07,857 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:07,857 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,857 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:07,857 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:07,857 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:07,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:07,858 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,858 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:07,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:07,858 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:07,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:07,858 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:07,858 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:07,858 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:07,858 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:07,859 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:07,859 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:07,859 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:07,859 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:07,859 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:07,859 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:07,859 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:07,859 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:07,859 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:07,859 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:07,859 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,859 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:07,859 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:07,860 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,860 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:07,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:07,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:07,860 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,860 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:07,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:07,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:07,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:07,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@768ad420[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:07,861 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:07,861 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,862 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,862 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:07,862 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:07,862 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:07,862 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,862 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:07,863 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,863 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:07,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:07,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:07,863 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:07,863 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:07,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:07,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:07,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:07,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:07,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:07,864 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:07,864 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:07,864 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:07,864 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:07,864 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:07,865 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:07,865 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:07,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,865 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:07,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:07,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:07,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:07,866 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:07,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,866 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:07,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,868 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,868 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:07,868 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:07,868 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:07,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:07,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:07,868 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:07,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:07,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:07,868 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:07,868 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:07,868 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:07,868 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,868 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,868 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,868 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:07,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:07,968 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:07,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:07,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:07,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:07,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:07,972 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:07,972 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:07,972 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:07,972 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:07,972 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,972 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:07,973 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,973 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:07,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:07,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:07,973 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,973 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:07,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:07,973 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:07,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:07,974 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:07,974 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:07,974 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:07,974 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:07,974 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:07,974 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:07,974 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:07,974 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:07,974 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:07,974 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:07,974 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:07,974 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:07,974 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:07,974 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:07,974 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,974 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:07,974 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:07,975 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,975 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:07,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:07,976 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,976 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:07,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:07,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:07,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:07,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:07,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11448057[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:07,977 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:07,977 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,977 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,977 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:07,977 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:07,977 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:07,977 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,978 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:07,978 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,978 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:07,978 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:07,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:07,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:07,978 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:07,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:07,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:07,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:07,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:07,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:07,980 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:07,980 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:07,980 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:07,980 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:07,980 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:07,980 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,980 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,981 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:07,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:07,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:07,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:07,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:07,981 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:07,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:07,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:07,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,982 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:07,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:07,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,983 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,983 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:07,983 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:07,983 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:07,983 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:07,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:07,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:07,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:07,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:07,983 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:07,983 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:07,984 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:07,984 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,984 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:07,984 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:07,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:07,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:07,984 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:07,984 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:08,019 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:08,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42018 deadline: 1681995428017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,019 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:08,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41978 deadline: 1681995428018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,021 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:08,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42008 deadline: 1681995428020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,022 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:08,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42076 deadline: 1681995428020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,022 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:08,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42016 deadline: 1681995428021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,084 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:08,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:08,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:08,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:08,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:08,086 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:08,086 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:08,086 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:08,086 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:08,087 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:08,087 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:08,087 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,087 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:08,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:08,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:08,087 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,087 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:08,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:08,088 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:08,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:08,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:08,088 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:08,088 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:08,088 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:08,088 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:08,088 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:08,088 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:08,088 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:08,088 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:08,088 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:08,088 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:08,089 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:08,089 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:08,089 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:08,089 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:08,089 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,089 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:08,089 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:08,090 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,090 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:08,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:08,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:08,091 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:08,091 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:08,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:08,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:08,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:08,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:08,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:08,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4574f82[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:08,092 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:08,092 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:08,092 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:08,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:08,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:08,092 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:08,092 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,092 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:08,092 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:08,092 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:08,093 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,093 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:08,093 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:08,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:08,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:08,093 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:08,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:08,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:08,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:08,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:08,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:08,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:08,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:08,095 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:08,095 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:08,095 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:08,095 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:08,095 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:08,096 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:08,096 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:08,096 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:08,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:08,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:08,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:08,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:08,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:08,096 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:08,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:08,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:08,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:08,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:08,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:08,097 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:08,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:08,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:08,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,099 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:08,099 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:08,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:08,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:08,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:08,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:08,099 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,099 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:08,099 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:08,099 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,099 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:08,099 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:08,099 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:08,099 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:08,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:08,099 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:08,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:08,100 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:08,100 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:08,142 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=22 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/B/a8b74ab8168d4f33bd1b3a2518d39221 2023-04-20 12:56:08,200 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:08,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:08,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:08,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:08,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:08,202 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:08,202 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:08,202 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:08,202 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:08,203 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:08,203 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:08,203 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:08,203 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:08,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:08,204 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,204 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:08,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:08,204 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:08,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:08,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:08,204 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:08,204 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:08,204 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:08,204 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:08,204 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:08,205 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:08,205 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:08,205 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:08,205 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:08,205 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:08,205 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:08,205 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:08,205 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:08,205 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:08,205 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,205 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:08,205 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:08,206 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,206 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:08,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:08,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:08,206 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:08,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:08,206 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:08,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:08,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:08,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:08,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:08,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32b9b59f[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:08,207 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:08,207 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:08,208 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:08,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:08,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:08,208 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:08,208 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:08,208 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,208 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:08,208 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:08,208 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,208 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:08,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:08,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:08,209 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:08,209 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:08,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:08,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:08,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:08,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:08,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:08,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:08,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:08,211 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:08,211 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:08,211 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:08,211 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:08,211 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:08,211 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:08,211 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:08,211 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:08,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:08,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:08,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:08,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:08,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:08,212 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:08,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:08,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:08,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:08,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:08,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:08,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:08,213 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:08,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:08,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,214 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:08,214 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:08,215 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:08,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:08,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:08,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:08,215 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:08,215 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:08,215 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:08,215 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:08,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:08,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:08,215 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:08,215 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,215 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:08,215 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:08,215 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:08,315 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:08,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:08,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:08,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:08,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:08,318 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:08,318 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:08,318 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:08,318 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:08,318 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:08,318 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:08,319 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,319 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:08,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:08,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:08,319 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,319 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:08,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:08,320 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:08,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:08,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:08,320 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:08,320 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:08,320 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:08,320 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:08,320 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:08,320 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:08,321 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:08,321 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:08,321 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:08,321 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:08,321 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:08,321 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:08,321 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:08,321 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:08,321 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,321 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:08,321 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:08,322 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:08,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:08,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:08,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:08,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:08,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:08,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:08,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12324c3c[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:08,323 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:08,323 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:08,324 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,324 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:08,326 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:08,328 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:08,328 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:08,328 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:08,328 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:08,328 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:08,328 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:08,329 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,329 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:08,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:08,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:08,329 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:08,329 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:08,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:08,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:08,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:08,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:08,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:08,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:08,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:08,330 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:08,330 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:08,330 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:08,331 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:08,331 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:08,331 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:08,331 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:08,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:08,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:08,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:08,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:08,331 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:08,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:08,332 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:08,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:08,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:08,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:08,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:08,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:08,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:08,333 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:08,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:08,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,334 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,334 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:08,334 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:08,334 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:08,334 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,334 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:08,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:08,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:08,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:08,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:08,334 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:08,334 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:08,334 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:08,334 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:08,334 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:08,334 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:08,334 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:08,334 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:08,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:08,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:08,434 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:08,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:08,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:08,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:08,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:08,439 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:08,439 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:08,439 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:08,439 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:08,439 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:08,439 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:08,440 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,440 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:08,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:08,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:08,441 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,441 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:08,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:08,441 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:08,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:08,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:08,441 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:08,449 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:08,449 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:08,449 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:08,449 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:08,449 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:08,450 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:08,450 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:08,450 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:08,450 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:08,450 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:08,450 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:08,450 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:08,450 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:08,450 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,450 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:08,450 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:08,455 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,455 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:08,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:08,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:08,455 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:08,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:08,455 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:08,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:08,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:08,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:08,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:08,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27412015[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:08,456 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:08,456 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:08,457 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:08,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:08,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:08,457 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:08,457 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:08,457 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:08,457 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,457 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:08,458 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:08,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:08,458 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:08,459 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:08,459 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:08,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:08,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:08,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:08,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:08,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:08,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:08,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:08,460 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:08,461 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:08,461 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:08,461 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:08,461 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:08,461 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:08,462 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:08,462 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:08,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:08,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:08,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:08,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:08,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:08,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:08,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:08,462 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:08,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:08,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:08,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:08,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:08,463 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:08,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:08,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,465 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:08,465 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:08,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:08,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:08,465 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:08,465 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:08,465 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:08,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:08,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:08,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:08,465 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:08,465 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:08,465 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:08,465 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,465 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:08,465 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:08,465 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:08,465 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:08,549 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=22 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/C/5d85ed4b5a7b4cf4916ec2c87c7c1d83 2023-04-20 12:56:08,552 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/A/8db65877342f457990d6b47df3d2be63 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/8db65877342f457990d6b47df3d2be63 2023-04-20 12:56:08,555 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/8db65877342f457990d6b47df3d2be63, entries=150, sequenceid=22, filesize=25.1 K 2023-04-20 12:56:08,556 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/B/a8b74ab8168d4f33bd1b3a2518d39221 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/a8b74ab8168d4f33bd1b3a2518d39221 2023-04-20 12:56:08,559 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/a8b74ab8168d4f33bd1b3a2518d39221, entries=150, sequenceid=22, filesize=11.6 K 2023-04-20 12:56:08,559 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/C/5d85ed4b5a7b4cf4916ec2c87c7c1d83 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/5d85ed4b5a7b4cf4916ec2c87c7c1d83 2023-04-20 12:56:08,562 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/5d85ed4b5a7b4cf4916ec2c87c7c1d83, entries=150, sequenceid=22, filesize=11.6 K 2023-04-20 12:56:08,562 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for 8113dc3e39e3420c86d60a8170c364f8 in 1667ms, sequenceid=22, compaction requested=false 2023-04-20 12:56:08,562 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8113dc3e39e3420c86d60a8170c364f8: 2023-04-20 12:56:08,565 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:08,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:08,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:08,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:08,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:08,570 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:08,570 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:08,570 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:08,570 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:08,570 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:08,570 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:08,571 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,571 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:08,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:08,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:08,571 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:08,571 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:08,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:08,571 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:08,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:08,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:08,572 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:08,572 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:08,572 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:08,572 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:08,572 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:08,572 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:08,572 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:08,572 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:08,572 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:08,572 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:08,572 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8113dc3e39e3420c86d60a8170c364f8 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-20 12:56:08,572 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=A 2023-04-20 12:56:08,573 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:08,573 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=B 2023-04-20 12:56:08,573 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:08,573 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=C 2023-04-20 12:56:08,573 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:08,589 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304201b0b5bd137814441b3bfcfd021299364 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304201b0b5bd137814441b3bfcfd021299364 2023-04-20 12:56:08,596 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=38, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/A/5121c5cbff4447f187104c5598bded90 2023-04-20 12:56:09,003 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=38 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/B/9bc0ae0820244d6a8a1947109cc06c70 2023-04-20 12:56:09,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 8113dc3e39e3420c86d60a8170c364f8 2023-04-20 12:56:09,025 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:09,033 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:09,034 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:09,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42016 deadline: 1681995429031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:09,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42018 deadline: 1681995429031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:09,034 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:09,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42008 deadline: 1681995429032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:09,034 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:09,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41978 deadline: 1681995429033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:09,034 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:09,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42076 deadline: 1681995429033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:09,090 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-20 12:56:09,136 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:09,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41978 deadline: 1681995429136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:09,137 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:09,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42008 deadline: 1681995429136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:09,137 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:09,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42076 deadline: 1681995429136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:09,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:09,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42018 deadline: 1681995429137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:09,138 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:09,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42016 deadline: 1681995429137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:09,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:09,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42008 deadline: 1681995429338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:09,340 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:09,340 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:09,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42076 deadline: 1681995429338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:09,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42018 deadline: 1681995429338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:09,340 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:09,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41978 deadline: 1681995429338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:09,340 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:09,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42016 deadline: 1681995429339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:09,411 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=38 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/C/fd4478ca22164caa9eecaa98a945d3d0 2023-04-20 12:56:09,414 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/A/5121c5cbff4447f187104c5598bded90 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/5121c5cbff4447f187104c5598bded90 2023-04-20 12:56:09,416 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/5121c5cbff4447f187104c5598bded90, entries=150, sequenceid=38, filesize=25.1 K 2023-04-20 12:56:09,417 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/B/9bc0ae0820244d6a8a1947109cc06c70 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/9bc0ae0820244d6a8a1947109cc06c70 2023-04-20 12:56:09,419 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/9bc0ae0820244d6a8a1947109cc06c70, entries=150, sequenceid=38, filesize=11.6 K 2023-04-20 12:56:09,420 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/C/fd4478ca22164caa9eecaa98a945d3d0 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/fd4478ca22164caa9eecaa98a945d3d0 2023-04-20 12:56:09,431 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/fd4478ca22164caa9eecaa98a945d3d0, entries=150, sequenceid=38, filesize=11.6 K 2023-04-20 12:56:09,432 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 8113dc3e39e3420c86d60a8170c364f8 in 860ms, sequenceid=38, compaction requested=false 2023-04-20 12:56:09,432 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8113dc3e39e3420c86d60a8170c364f8: 2023-04-20 12:56:09,432 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:09,432 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:09,432 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:09,432 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:09,434 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:09,434 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:09,435 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:09,435 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:09,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:09,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:09,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:09,435 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:09,435 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:09,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:09,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:09,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:09,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:09,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:09,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:09,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19544020[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:09,437 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:09,437 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:09,438 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:09,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:09,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:09,438 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:09,438 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:09,438 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:09,438 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:09,438 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:09,438 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:09,438 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:09,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:09,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:09,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:09,439 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:09,439 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:09,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:09,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:09,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:09,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:09,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:09,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:09,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:09,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:09,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:09,440 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:09,440 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:09,440 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:09,440 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:09,440 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:09,441 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:09,441 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:09,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:09,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:09,441 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:09,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:09,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:09,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:09,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:09,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:09,442 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:09,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:09,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:09,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:09,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:09,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:09,443 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:09,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:09,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:09,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:09,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:09,444 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:09,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:09,444 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:09,444 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:09,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:09,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:09,444 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:09,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:09,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:09,444 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:09,444 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:09,444 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:09,444 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:09,445 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:09,445 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:09,445 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:09,445 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:09,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:09,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:09,545 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:09,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:09,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:09,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:09,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:09,547 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:09,547 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:09,547 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:09,547 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:09,547 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:09,547 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:09,548 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:09,548 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:09,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:09,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:09,548 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:09,548 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:09,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:09,549 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:09,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:09,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:09,549 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:09,549 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:09,549 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:09,549 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:09,549 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:09,550 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:09,550 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:09,550 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:09,550 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:09,550 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:09,550 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8113dc3e39e3420c86d60a8170c364f8 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-20 12:56:09,550 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=A 2023-04-20 12:56:09,550 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:09,550 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=B 2023-04-20 12:56:09,550 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:09,550 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=C 2023-04-20 12:56:09,550 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:09,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 8113dc3e39e3420c86d60a8170c364f8 2023-04-20 12:56:09,645 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:09,654 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:09,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42016 deadline: 1681995429651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:09,656 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:09,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42008 deadline: 1681995429653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:09,656 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:09,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41978 deadline: 1681995429653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:09,656 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:09,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42018 deadline: 1681995429653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:09,656 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:09,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42076 deadline: 1681995429654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:09,756 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:09,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42016 deadline: 1681995429755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:09,758 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:09,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42008 deadline: 1681995429757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:09,758 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:09,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41978 deadline: 1681995429757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:09,758 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:09,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42018 deadline: 1681995429757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:09,758 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:09,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42076 deadline: 1681995429757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:09,959 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:09,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42016 deadline: 1681995429958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:09,960 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:09,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42008 deadline: 1681995429959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:09,960 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:09,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41978 deadline: 1681995429959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:09,960 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420997f9918d554485c816fad6bfd25dfc8 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420997f9918d554485c816fad6bfd25dfc8 2023-04-20 12:56:09,960 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:09,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42076 deadline: 1681995429959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:09,963 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:09,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42018 deadline: 1681995429963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:09,964 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=54, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/A/a7107293519240149a6f840406885f2e 2023-04-20 12:56:10,262 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:10,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42008 deadline: 1681995430261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:10,263 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:10,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42076 deadline: 1681995430261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:10,263 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:10,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42016 deadline: 1681995430262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:10,264 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:10,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41978 deadline: 1681995430262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:10,269 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:10,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42018 deadline: 1681995430266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:10,375 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=54 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/B/c2a8189a8fee4afbaa5980026c54d782 2023-04-20 12:56:10,421 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=54 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/C/0c8e6efed97a44658ef7188ab811de74 2023-04-20 12:56:10,426 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/A/a7107293519240149a6f840406885f2e as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/a7107293519240149a6f840406885f2e 2023-04-20 12:56:10,430 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/a7107293519240149a6f840406885f2e, entries=150, sequenceid=54, filesize=25.1 K 2023-04-20 12:56:10,431 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/B/c2a8189a8fee4afbaa5980026c54d782 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/c2a8189a8fee4afbaa5980026c54d782 2023-04-20 12:56:10,436 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/c2a8189a8fee4afbaa5980026c54d782, entries=150, sequenceid=54, filesize=11.6 K 2023-04-20 12:56:10,437 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/C/0c8e6efed97a44658ef7188ab811de74 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/0c8e6efed97a44658ef7188ab811de74 2023-04-20 12:56:10,447 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/0c8e6efed97a44658ef7188ab811de74, entries=150, sequenceid=54, filesize=11.6 K 2023-04-20 12:56:10,448 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for 8113dc3e39e3420c86d60a8170c364f8 in 898ms, sequenceid=54, compaction requested=true 2023-04-20 12:56:10,448 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8113dc3e39e3420c86d60a8170c364f8: 2023-04-20 12:56:10,448 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:10,448 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:10,448 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:10,448 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:10,448 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:10,448 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:10,449 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:10,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:10,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:10,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:10,449 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:10,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:10,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:10,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:10,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:10,450 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:10,450 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:10,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:10,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:10,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66be0bd8[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:10,450 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:10,450 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:10,451 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:10,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:10,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:10,451 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:10,451 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:10,451 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:10,452 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:10,452 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:10,452 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:10,452 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:10,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:10,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:10,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:10,452 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:10,452 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:10,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:10,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:10,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:10,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:10,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:10,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:10,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:10,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:10,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:10,455 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:10,455 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:10,455 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:10,455 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:10,455 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:10,456 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:10,456 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:10,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:10,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:10,456 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:10,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:10,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:10,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:10,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:10,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:10,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:10,457 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:10,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:10,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:10,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:10,459 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:10,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:10,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:10,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:10,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:10,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:10,460 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:10,460 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:10,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:10,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:10,461 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:10,461 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:10,461 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:10,461 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:10,461 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:10,461 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:10,461 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:10,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:10,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:10,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:10,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:10,461 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:10,461 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:10,461 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:10,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:10,561 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:10,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:10,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:10,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:10,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:10,564 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:10,564 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:10,564 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:10,564 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:10,565 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:10,565 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:10,565 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:10,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:10,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:10,566 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:10,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:10,566 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:10,566 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:10,566 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:10,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:10,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:10,566 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:10,567 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:10,567 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:10,567 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:10,567 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:10,567 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:10,567 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:10,567 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:10,567 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:10,567 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:10,567 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8113dc3e39e3420c86d60a8170c364f8 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-20 12:56:10,568 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=A 2023-04-20 12:56:10,568 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:10,568 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=B 2023-04-20 12:56:10,568 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:10,568 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=C 2023-04-20 12:56:10,568 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:10,586 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304200729156cfe6f46428f39fe76cfa7a3dc to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304200729156cfe6f46428f39fe76cfa7a3dc 2023-04-20 12:56:10,591 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=71, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/A/00b4802d234046e894dc99af2e9e491d 2023-04-20 12:56:10,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 8113dc3e39e3420c86d60a8170c364f8 2023-04-20 12:56:10,766 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:10,799 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:10,799 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:10,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42008 deadline: 1681995430794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:10,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41978 deadline: 1681995430794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:10,799 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:10,799 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:10,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42016 deadline: 1681995430795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:10,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42076 deadline: 1681995430794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:10,799 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:10,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42018 deadline: 1681995430795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:10,900 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:10,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42008 deadline: 1681995430900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:10,901 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:10,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41978 deadline: 1681995430900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:10,901 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:10,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42016 deadline: 1681995430900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:10,901 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:10,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42076 deadline: 1681995430900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:10,901 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:10,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42018 deadline: 1681995430900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:11,014 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=71 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/B/c6e30f105f2540e3a16aab5c3db2eeb4 2023-04-20 12:56:11,102 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:11,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42008 deadline: 1681995431101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:11,103 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:11,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41978 deadline: 1681995431101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:11,103 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:11,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42016 deadline: 1681995431102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:11,104 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:11,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42076 deadline: 1681995431103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:11,104 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:11,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42018 deadline: 1681995431103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:11,404 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:11,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41978 deadline: 1681995431403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:11,405 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:11,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42008 deadline: 1681995431404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:11,406 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:11,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42018 deadline: 1681995431405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:11,406 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:11,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42076 deadline: 1681995431405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:11,407 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:11,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42016 deadline: 1681995431405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:11,421 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=71 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/C/ac33a56c447f4806bab626b20fdcf70f 2023-04-20 12:56:11,424 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/A/00b4802d234046e894dc99af2e9e491d as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/00b4802d234046e894dc99af2e9e491d 2023-04-20 12:56:11,427 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/00b4802d234046e894dc99af2e9e491d, entries=150, sequenceid=71, filesize=25.1 K 2023-04-20 12:56:11,428 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/B/c6e30f105f2540e3a16aab5c3db2eeb4 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/c6e30f105f2540e3a16aab5c3db2eeb4 2023-04-20 12:56:11,430 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/c6e30f105f2540e3a16aab5c3db2eeb4, entries=150, sequenceid=71, filesize=11.6 K 2023-04-20 12:56:11,431 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/C/ac33a56c447f4806bab626b20fdcf70f as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/ac33a56c447f4806bab626b20fdcf70f 2023-04-20 12:56:11,434 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/ac33a56c447f4806bab626b20fdcf70f, entries=150, sequenceid=71, filesize=11.6 K 2023-04-20 12:56:11,434 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 8113dc3e39e3420c86d60a8170c364f8 in 867ms, sequenceid=71, compaction requested=true 2023-04-20 12:56:11,434 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8113dc3e39e3420c86d60a8170c364f8: 2023-04-20 12:56:11,435 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:11,435 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:11,435 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:11,435 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:11,435 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:11,435 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:11,436 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:11,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:11,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:11,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:11,436 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:11,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:11,436 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:11,436 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:11,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:11,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:11,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:11,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:11,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:11,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66edf89f[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:11,437 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:11,437 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:11,437 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:11,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:11,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:11,438 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:11,438 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:11,438 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:11,438 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:11,438 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:11,438 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:11,438 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:11,438 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:11,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:11,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:11,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:11,438 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:11,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:11,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:11,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:11,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:11,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:11,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:11,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:11,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:11,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:11,440 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:11,440 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:11,440 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:11,440 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:11,440 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:11,443 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:11,443 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:11,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:11,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:11,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:11,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:11,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:11,443 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:11,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:11,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:11,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:11,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:11,443 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:11,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:11,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:11,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:11,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:11,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:11,444 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:11,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:11,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:11,446 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:11,446 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:11,446 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:11,446 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:11,446 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:11,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:11,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:11,446 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:11,446 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:11,446 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:11,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:11,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:11,446 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:11,446 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:11,446 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:11,446 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:11,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:11,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:11,547 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:11,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:11,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:11,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:11,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:11,549 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:11,549 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:11,549 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:11,549 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:11,549 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:11,549 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:11,550 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:11,550 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:11,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:11,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:11,550 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:11,550 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:11,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:11,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:11,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:11,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:11,551 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:11,552 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:11,552 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:11,552 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:11,552 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:11,552 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:11,553 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:11,553 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:11,553 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:11,553 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:11,553 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8113dc3e39e3420c86d60a8170c364f8 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-20 12:56:11,553 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=A 2023-04-20 12:56:11,553 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:11,553 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=B 2023-04-20 12:56:11,553 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:11,553 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=C 2023-04-20 12:56:11,553 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:11,562 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420760b394812d941928c4224da5ff0a6e5 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420760b394812d941928c4224da5ff0a6e5 2023-04-20 12:56:11,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 8113dc3e39e3420c86d60a8170c364f8 2023-04-20 12:56:11,908 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:11,923 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:11,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42018 deadline: 1681995431919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:11,923 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:11,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42076 deadline: 1681995431921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:11,926 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:11,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41978 deadline: 1681995431922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:11,926 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:11,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42016 deadline: 1681995431922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:11,927 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:11,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42008 deadline: 1681995431923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:11,966 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=86, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/A/d07c24c693cb4f71a50db907611b70a9 2023-04-20 12:56:12,025 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:12,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42018 deadline: 1681995432024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,026 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:12,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42076 deadline: 1681995432024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,030 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:12,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41978 deadline: 1681995432027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,030 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:12,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42016 deadline: 1681995432027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,030 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:12,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42008 deadline: 1681995432027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,228 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:12,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42018 deadline: 1681995432226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,229 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:12,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42076 deadline: 1681995432228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,232 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:12,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41978 deadline: 1681995432230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,233 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:12,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42008 deadline: 1681995432231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,233 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:12,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42016 deadline: 1681995432232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,382 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=86 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/B/f04ab1daa88445b9a38f487cc3fe6725 2023-04-20 12:56:12,390 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=86 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/C/2e46722a70684cecb1d46eccdb40343d 2023-04-20 12:56:12,393 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/A/d07c24c693cb4f71a50db907611b70a9 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/d07c24c693cb4f71a50db907611b70a9 2023-04-20 12:56:12,396 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/d07c24c693cb4f71a50db907611b70a9, entries=150, sequenceid=86, filesize=25.1 K 2023-04-20 12:56:12,397 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/B/f04ab1daa88445b9a38f487cc3fe6725 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/f04ab1daa88445b9a38f487cc3fe6725 2023-04-20 12:56:12,399 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/f04ab1daa88445b9a38f487cc3fe6725, entries=150, sequenceid=86, filesize=11.6 K 2023-04-20 12:56:12,400 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/C/2e46722a70684cecb1d46eccdb40343d as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/2e46722a70684cecb1d46eccdb40343d 2023-04-20 12:56:12,403 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/2e46722a70684cecb1d46eccdb40343d, entries=150, sequenceid=86, filesize=11.6 K 2023-04-20 12:56:12,404 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for 8113dc3e39e3420c86d60a8170c364f8 in 851ms, sequenceid=86, compaction requested=true 2023-04-20 12:56:12,404 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8113dc3e39e3420c86d60a8170c364f8: 2023-04-20 12:56:12,404 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:12,404 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:12,404 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:12,404 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,404 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:12,404 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:12,405 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,405 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:12,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:12,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:12,405 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:12,405 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:12,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:12,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:12,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:12,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:12,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:12,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48023ddf[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:12,406 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:12,406 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:12,406 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:12,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:12,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:12,407 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:12,407 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:12,407 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:12,407 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,407 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:12,407 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:12,407 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,407 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:12,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:12,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:12,407 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:12,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:12,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:12,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:12,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:12,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:12,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:12,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:12,409 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:12,409 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:12,409 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:12,409 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:12,409 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:12,410 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:12,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:12,410 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:12,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:12,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:12,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:12,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:12,410 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:12,410 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:12,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:12,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:12,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:12,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:12,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:12,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:12,411 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:12,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:12,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,412 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,412 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:12,412 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:12,412 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:12,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:12,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:12,412 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:12,412 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:12,412 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:12,412 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:12,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:12,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:12,412 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:12,412 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,412 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:12,412 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:12,412 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:12,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:12,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:12,512 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:12,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:12,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:12,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:12,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:12,521 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:12,521 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:12,521 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:12,521 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:12,522 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:12,522 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:12,523 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:12,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:12,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:12,523 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:12,523 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,523 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:12,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:12,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:12,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:12,524 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:12,524 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:12,524 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:12,524 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:12,524 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:12,524 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:12,524 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:12,524 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:12,524 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:12,525 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:12,525 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8113dc3e39e3420c86d60a8170c364f8 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-20 12:56:12,525 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=A 2023-04-20 12:56:12,525 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:12,525 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=B 2023-04-20 12:56:12,525 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:12,525 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=C 2023-04-20 12:56:12,525 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:12,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 8113dc3e39e3420c86d60a8170c364f8 2023-04-20 12:56:12,533 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:12,542 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304207da0196c58694ae9a26388df7db0900c to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304207da0196c58694ae9a26388df7db0900c 2023-04-20 12:56:12,546 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:12,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42016 deadline: 1681995432540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,547 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=104, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/A/6c4802930e584eb1bf31e4b8f091bdb8 2023-04-20 12:56:12,549 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:12,549 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:12,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42076 deadline: 1681995432545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41978 deadline: 1681995432545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,549 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:12,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42008 deadline: 1681995432546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,549 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:12,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42018 deadline: 1681995432546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,556 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=104 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/B/3252163496a3495cae7662d0aa46ab38 2023-04-20 12:56:12,564 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=104 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/C/89a20537a1824ef7b5c4fd5cc79e351a 2023-04-20 12:56:12,568 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/A/6c4802930e584eb1bf31e4b8f091bdb8 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/6c4802930e584eb1bf31e4b8f091bdb8 2023-04-20 12:56:12,570 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/6c4802930e584eb1bf31e4b8f091bdb8, entries=150, sequenceid=104, filesize=25.1 K 2023-04-20 12:56:12,571 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/B/3252163496a3495cae7662d0aa46ab38 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/3252163496a3495cae7662d0aa46ab38 2023-04-20 12:56:12,573 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/3252163496a3495cae7662d0aa46ab38, entries=150, sequenceid=104, filesize=11.6 K 2023-04-20 12:56:12,574 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/C/89a20537a1824ef7b5c4fd5cc79e351a as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/89a20537a1824ef7b5c4fd5cc79e351a 2023-04-20 12:56:12,576 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/89a20537a1824ef7b5c4fd5cc79e351a, entries=150, sequenceid=104, filesize=11.6 K 2023-04-20 12:56:12,577 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 8113dc3e39e3420c86d60a8170c364f8 in 52ms, sequenceid=104, compaction requested=true 2023-04-20 12:56:12,577 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8113dc3e39e3420c86d60a8170c364f8: 2023-04-20 12:56:12,577 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:12,577 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:12,577 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:12,577 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,577 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:12,577 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:12,578 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,578 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:12,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:12,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:12,578 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:12,578 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:12,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:12,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:12,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:12,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:12,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:12,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@736c31bf[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:12,580 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:12,580 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:12,580 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:12,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:12,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:12,580 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:12,580 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:12,580 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:12,580 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,581 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:12,581 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,581 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:12,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:12,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:12,581 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:12,581 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:12,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:12,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:12,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:12,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:12,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:12,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:12,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:12,583 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:12,583 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:12,583 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:12,583 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:12,583 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:12,584 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:12,584 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:12,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:12,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:12,584 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:12,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:12,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:12,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:12,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:12,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:12,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:12,585 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:12,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:12,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:12,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:12,586 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:12,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:12,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,587 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:12,587 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:12,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:12,587 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:12,587 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:12,587 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:12,587 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:12,587 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:12,587 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:12,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:12,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:12,587 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:12,588 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,588 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:12,588 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:12,588 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:12,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:12,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:12,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 8113dc3e39e3420c86d60a8170c364f8 2023-04-20 12:56:12,649 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8113dc3e39e3420c86d60a8170c364f8 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-20 12:56:12,649 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=A 2023-04-20 12:56:12,649 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:12,649 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=B 2023-04-20 12:56:12,649 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:12,649 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=C 2023-04-20 12:56:12,649 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:12,658 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420b052abf9e09542d3bd94148b71708ad7 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420b052abf9e09542d3bd94148b71708ad7 2023-04-20 12:56:12,670 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:12,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41978 deadline: 1681995432665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,671 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:12,671 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:12,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42076 deadline: 1681995432666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42008 deadline: 1681995432666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,671 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:12,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42016 deadline: 1681995432666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:12,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42018 deadline: 1681995432670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,688 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:12,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:12,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:12,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:12,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:12,691 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:12,691 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:12,691 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:12,691 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:12,691 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:12,691 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:12,692 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:12,692 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:12,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:12,692 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,692 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:12,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:12,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:12,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:12,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:12,693 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:12,693 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:12,693 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:12,693 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:12,693 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:12,693 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:12,693 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:12,693 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:12,693 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:12,693 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:12,693 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:12,693 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:12,693 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:12,693 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:12,693 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,693 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:12,693 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:12,694 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,694 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:12,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:12,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:12,694 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:12,694 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:12,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:12,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:12,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:12,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:12,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:12,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@651a04c1[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:12,695 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:12,695 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:12,696 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:12,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:12,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:12,696 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:12,696 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,696 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:12,696 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:12,696 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:12,697 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:12,697 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:12,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:12,697 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:12,697 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:12,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:12,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:12,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:12,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:12,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:12,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:12,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:12,698 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:12,698 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:12,698 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:12,698 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:12,698 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:12,699 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:12,699 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:12,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:12,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:12,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:12,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:12,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:12,699 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:12,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:12,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:12,699 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:12,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:12,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:12,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:12,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:12,700 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:12,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:12,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,702 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,702 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:12,702 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:12,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:12,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:12,702 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:12,702 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:12,702 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:12,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:12,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:12,702 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,702 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:12,703 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:12,702 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:12,703 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:12,703 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:12,703 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:12,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:12,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:12,774 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:12,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41978 deadline: 1681995432771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,774 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:12,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42076 deadline: 1681995432772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,774 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:12,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42008 deadline: 1681995432772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,774 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:12,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42016 deadline: 1681995432772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,777 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:12,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42018 deadline: 1681995432775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,803 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:12,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:12,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:12,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:12,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:12,806 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:12,806 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:12,806 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:12,806 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:12,806 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:12,806 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:12,807 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,807 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:12,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:12,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:12,807 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,807 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:12,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:12,808 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:12,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:12,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:12,808 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:12,808 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:12,808 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:12,808 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:12,808 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:12,808 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:12,808 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:12,808 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:12,808 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:12,808 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:12,809 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:12,809 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:12,809 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:12,809 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:12,809 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,809 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:12,809 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:12,809 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:12,809 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:12,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:12,810 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:12,810 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:12,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:12,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:12,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:12,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:12,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:12,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c3a21fd[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:12,811 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:12,811 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:12,811 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:12,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:12,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:12,811 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,812 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:12,811 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:12,812 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:12,812 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:12,812 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,812 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:12,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:12,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:12,812 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:12,812 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:12,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:12,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:12,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:12,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:12,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:12,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:12,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:12,814 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:12,814 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:12,814 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:12,814 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:12,814 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:12,815 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:12,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:12,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:12,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:12,815 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:12,815 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:12,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:12,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:12,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:12,815 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:12,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:12,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:12,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:12,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:12,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:12,816 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:12,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:12,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,817 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:12,818 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:12,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:12,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:12,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:12,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:12,818 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,818 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:12,818 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:12,818 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,818 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:12,818 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:12,818 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:12,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:12,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:12,818 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:12,818 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:12,818 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:12,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:12,918 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:12,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:12,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:12,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:12,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:12,921 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:12,921 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:12,921 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:12,921 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:12,922 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:12,922 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:12,922 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,922 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:12,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:12,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:12,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:12,923 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,923 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:12,923 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:12,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:12,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:12,923 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:12,923 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:12,923 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:12,923 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:12,923 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:12,923 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:12,924 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:12,924 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:12,924 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:12,924 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:12,924 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:12,924 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:12,924 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:12,924 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:12,924 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,924 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:12,924 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:12,925 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,925 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:12,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:12,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:12,925 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:12,925 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:12,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:12,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:12,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:12,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:12,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:12,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@350f0560[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:12,926 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:12,926 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:12,927 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:12,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:12,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:12,927 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:12,927 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,927 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:12,927 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:12,927 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:12,928 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,928 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:12,928 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:12,928 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:12,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:12,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:12,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:12,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:12,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:12,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:12,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:12,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:12,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:12,930 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:12,930 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:12,930 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:12,930 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:12,930 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:12,931 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:12,931 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:12,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:12,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:12,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:12,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:12,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:12,931 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:12,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:12,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:12,931 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:12,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:12,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:12,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:12,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:12,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,932 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:12,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:12,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,933 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,934 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:12,934 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:12,934 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:12,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:12,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:12,934 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:12,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:12,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:12,934 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:12,934 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:12,934 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:12,934 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:12,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:12,934 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:12,934 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:12,934 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:12,976 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:12,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42008 deadline: 1681995432975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,976 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:12,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42016 deadline: 1681995432975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,977 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:12,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41978 deadline: 1681995432975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,977 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:12,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42076 deadline: 1681995432975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:12,980 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:12,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42018 deadline: 1681995432978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,034 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:13,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:13,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:13,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:13,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:13,037 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:13,037 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:13,037 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:13,037 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:13,037 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:13,037 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:13,038 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:13,038 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:13,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:13,038 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,038 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:13,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:13,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:13,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:13,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:13,039 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:13,039 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:13,039 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:13,039 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:13,039 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:13,039 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:13,039 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:13,039 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:13,039 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:13,039 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:13,040 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:13,040 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:13,040 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:13,040 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:13,040 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,040 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:13,040 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:13,041 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,041 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:13,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:13,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:13,041 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:13,041 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:13,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:13,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:13,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:13,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:13,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:13,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@277c3bb7[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:13,042 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:13,042 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:13,043 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:13,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:13,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:13,043 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:13,043 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:13,043 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,043 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:13,043 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:13,043 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:13,044 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:13,044 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,044 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:13,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:13,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:13,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:13,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:13,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:13,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:13,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:13,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:13,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:13,045 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:13,046 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:13,046 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:13,046 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:13,046 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:13,046 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:13,046 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:13,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:13,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:13,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:13,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:13,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:13,046 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:13,047 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:13,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:13,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:13,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:13,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:13,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:13,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:13,048 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:13,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:13,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,049 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:13,049 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,049 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:13,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:13,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:13,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:13,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:13,049 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:13,050 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:13,049 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:13,050 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:13,050 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,050 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:13,050 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:13,050 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:13,050 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:13,050 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:13,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:13,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:13,062 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=119, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/A/bef4c7402e3945f589b20ce014cc9974 2023-04-20 12:56:13,069 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=119 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/B/a381f1c4edda40559141761107fd7e01 2023-04-20 12:56:13,150 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:13,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:13,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:13,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:13,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:13,162 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:13,162 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:13,162 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:13,162 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:13,162 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:13,162 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:13,163 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,163 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:13,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:13,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:13,163 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,163 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:13,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:13,164 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:13,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:13,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:13,164 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:13,164 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:13,164 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:13,164 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:13,164 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:13,164 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:13,164 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:13,164 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:13,164 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:13,165 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:13,165 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:13,165 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:13,165 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:13,165 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,165 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:13,165 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:13,165 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,165 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:13,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:13,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:13,166 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:13,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:13,166 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:13,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:13,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:13,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:13,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:13,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12f50f95[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:13,167 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:13,167 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:13,167 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:13,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:13,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:13,167 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:13,168 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:13,167 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,168 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:13,168 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:13,168 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:13,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:13,168 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:13,168 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:13,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:13,169 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:13,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:13,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:13,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:13,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:13,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:13,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:13,170 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:13,170 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:13,170 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:13,170 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:13,170 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:13,171 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:13,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:13,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:13,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:13,171 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:13,171 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:13,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:13,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:13,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:13,171 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:13,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:13,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:13,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:13,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:13,172 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:13,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:13,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:13,173 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,173 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:13,174 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:13,174 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,174 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:13,174 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:13,174 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:13,174 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:13,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:13,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:13,174 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:13,174 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:13,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:13,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:13,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:13,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:13,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:13,176 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:13,176 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:13,276 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:13,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:13,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:13,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:13,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:13,279 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:13,279 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:13,279 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:13,279 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:13,279 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:13,279 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:13,279 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:13,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42016 deadline: 1681995433277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,280 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:13,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42008 deadline: 1681995433277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,280 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,280 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:13,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:13,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:13,280 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,280 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:13,280 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:13,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41978 deadline: 1681995433278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:13,280 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:13,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42076 deadline: 1681995433278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,281 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:13,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:13,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:13,281 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:13,281 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:13,281 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:13,281 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:13,281 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:13,281 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:13,281 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:13,281 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:13,281 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:13,282 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:13,282 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:13,282 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:13,282 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:13,282 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:13,282 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,282 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:13,282 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:13,283 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:13,283 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:13,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:13,283 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:13,283 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:13,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:13,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:13,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:13,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:13,284 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:13,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:13,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42018 deadline: 1681995433281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,284 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:13,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f7de1ff[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:13,284 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:13,285 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:13,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:13,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:13,285 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:13,285 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:13,285 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,285 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:13,285 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:13,285 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:13,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:13,286 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:13,286 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:13,286 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:13,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:13,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:13,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:13,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:13,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:13,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:13,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:13,287 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:13,287 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:13,287 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:13,287 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:13,287 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:13,288 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:13,288 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:13,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:13,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:13,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:13,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:13,288 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:13,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:13,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:13,289 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:13,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:13,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:13,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:13,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:13,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:13,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:13,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:13,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,294 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,295 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,295 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:13,295 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:13,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:13,296 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:13,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:13,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:13,296 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:13,296 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,296 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:13,296 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:13,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:13,296 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:13,296 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:13,296 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:13,296 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:13,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:13,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:13,396 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:13,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:13,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:13,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:13,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:13,399 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:13,399 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:13,399 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:13,399 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:13,399 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:13,399 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:13,400 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,400 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:13,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:13,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:13,400 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,400 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:13,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:13,400 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:13,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:13,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:13,400 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:13,401 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:13,401 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:13,401 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:13,401 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:13,401 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:13,401 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:13,401 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:13,401 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:13,401 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:13,401 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:13,401 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:13,401 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:13,401 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:13,401 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,401 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:13,401 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:13,402 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,402 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:13,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:13,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:13,402 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:13,402 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:13,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:13,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:13,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:13,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:13,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:13,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6294c575[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:13,403 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:13,403 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:13,404 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:13,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:13,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:13,404 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,404 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:13,404 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:13,404 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:13,404 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:13,404 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,404 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:13,404 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:13,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:13,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:13,404 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:13,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:13,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:13,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:13,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:13,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:13,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:13,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:13,406 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:13,406 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:13,406 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:13,406 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:13,406 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:13,407 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:13,407 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:13,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:13,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:13,407 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:13,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:13,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:13,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:13,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:13,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:13,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:13,408 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:13,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:13,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:13,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:13,408 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:13,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:13,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,409 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:13,409 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:13,409 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:13,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:13,409 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:13,410 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:13,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:13,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:13,409 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:13,410 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,410 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:13,410 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:13,410 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:13,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:13,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:13,410 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:13,410 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:13,410 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:13,476 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=119 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/C/3b95b18cd7194c839aa240eae38ed958 2023-04-20 12:56:13,480 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/A/bef4c7402e3945f589b20ce014cc9974 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/bef4c7402e3945f589b20ce014cc9974 2023-04-20 12:56:13,485 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/bef4c7402e3945f589b20ce014cc9974, entries=150, sequenceid=119, filesize=25.1 K 2023-04-20 12:56:13,486 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/B/a381f1c4edda40559141761107fd7e01 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/a381f1c4edda40559141761107fd7e01 2023-04-20 12:56:13,490 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/a381f1c4edda40559141761107fd7e01, entries=150, sequenceid=119, filesize=11.6 K 2023-04-20 12:56:13,491 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/C/3b95b18cd7194c839aa240eae38ed958 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/3b95b18cd7194c839aa240eae38ed958 2023-04-20 12:56:13,501 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/3b95b18cd7194c839aa240eae38ed958, entries=150, sequenceid=119, filesize=11.6 K 2023-04-20 12:56:13,502 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 8113dc3e39e3420c86d60a8170c364f8 in 854ms, sequenceid=119, compaction requested=true 2023-04-20 12:56:13,502 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8113dc3e39e3420c86d60a8170c364f8: 2023-04-20 12:56:13,502 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:56:13,502 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-20 12:56:13,502 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-20 12:56:13,502 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:56:13,502 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:56:13,504 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 180222 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-20 12:56:13,504 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 83202 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-20 12:56:13,504 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): 8113dc3e39e3420c86d60a8170c364f8/B is initiating minor compaction (all files) 2023-04-20 12:56:13,504 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): 8113dc3e39e3420c86d60a8170c364f8/A is initiating minor compaction (all files) 2023-04-20 12:56:13,505 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8113dc3e39e3420c86d60a8170c364f8/B in TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:13,505 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8113dc3e39e3420c86d60a8170c364f8/A in TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:13,505 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/8db65877342f457990d6b47df3d2be63, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/5121c5cbff4447f187104c5598bded90, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/a7107293519240149a6f840406885f2e, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/00b4802d234046e894dc99af2e9e491d, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/d07c24c693cb4f71a50db907611b70a9, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/6c4802930e584eb1bf31e4b8f091bdb8, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/bef4c7402e3945f589b20ce014cc9974] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp, totalSize=176.0 K 2023-04-20 12:56:13,505 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/a8b74ab8168d4f33bd1b3a2518d39221, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/9bc0ae0820244d6a8a1947109cc06c70, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/c2a8189a8fee4afbaa5980026c54d782, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/c6e30f105f2540e3a16aab5c3db2eeb4, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/f04ab1daa88445b9a38f487cc3fe6725, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/3252163496a3495cae7662d0aa46ab38, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/a381f1c4edda40559141761107fd7e01] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp, totalSize=81.3 K 2023-04-20 12:56:13,505 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 8db65877342f457990d6b47df3d2be63, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=22, earliestPutTs=1681995366895 2023-04-20 12:56:13,505 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting a8b74ab8168d4f33bd1b3a2518d39221, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=22, earliestPutTs=1681995366895 2023-04-20 12:56:13,505 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 5121c5cbff4447f187104c5598bded90, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=38, earliestPutTs=1681995366899 2023-04-20 12:56:13,505 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 9bc0ae0820244d6a8a1947109cc06c70, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=38, earliestPutTs=1681995366899 2023-04-20 12:56:13,505 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting a7107293519240149a6f840406885f2e, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=54, earliestPutTs=1681995369031 2023-04-20 12:56:13,506 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting c2a8189a8fee4afbaa5980026c54d782, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=54, earliestPutTs=1681995369031 2023-04-20 12:56:13,506 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 00b4802d234046e894dc99af2e9e491d, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1681995369652 2023-04-20 12:56:13,506 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting c6e30f105f2540e3a16aab5c3db2eeb4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1681995369652 2023-04-20 12:56:13,506 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting d07c24c693cb4f71a50db907611b70a9, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1681995370770 2023-04-20 12:56:13,506 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting f04ab1daa88445b9a38f487cc3fe6725, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1681995370770 2023-04-20 12:56:13,506 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 6c4802930e584eb1bf31e4b8f091bdb8, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=104, earliestPutTs=1681995371918 2023-04-20 12:56:13,507 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 3252163496a3495cae7662d0aa46ab38, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=104, earliestPutTs=1681995371918 2023-04-20 12:56:13,507 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting bef4c7402e3945f589b20ce014cc9974, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=119, earliestPutTs=1681995372539 2023-04-20 12:56:13,507 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting a381f1c4edda40559141761107fd7e01, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=119, earliestPutTs=1681995372539 2023-04-20 12:56:13,510 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:13,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:13,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:13,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:13,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:13,514 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:13,514 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:13,514 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:13,514 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:13,514 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:13,514 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:13,515 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,515 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:13,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:13,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:13,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:13,516 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,516 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:13,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:13,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:13,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:13,516 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:13,517 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:13,517 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:13,517 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:13,517 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:13,517 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:13,518 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:13,518 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:13,518 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:13,518 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:13,518 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8113dc3e39e3420c86d60a8170c364f8 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-20 12:56:13,518 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=A 2023-04-20 12:56:13,519 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:13,519 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=B 2023-04-20 12:56:13,519 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:13,519 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=C 2023-04-20 12:56:13,519 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:13,519 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): 8113dc3e39e3420c86d60a8170c364f8#B#compaction#502 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:56:13,560 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8113dc3e39e3420c86d60a8170c364f8#A#compaction#503 average throughput is 0.58 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:56:13,573 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/B/5ddb5df9bd9c4ecdba8287d3ee2cbfca as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/5ddb5df9bd9c4ecdba8287d3ee2cbfca 2023-04-20 12:56:13,582 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 8113dc3e39e3420c86d60a8170c364f8/B of 8113dc3e39e3420c86d60a8170c364f8 into 5ddb5df9bd9c4ecdba8287d3ee2cbfca(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:56:13,582 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8113dc3e39e3420c86d60a8170c364f8: 2023-04-20 12:56:13,582 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8., storeName=8113dc3e39e3420c86d60a8170c364f8/B, priority=9, startTime=1681995373502; duration=0sec 2023-04-20 12:56:13,583 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:56:13,583 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-20 12:56:13,584 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 83202 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-20 12:56:13,584 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): 8113dc3e39e3420c86d60a8170c364f8/C is initiating minor compaction (all files) 2023-04-20 12:56:13,584 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8113dc3e39e3420c86d60a8170c364f8/C in TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:13,584 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/5d85ed4b5a7b4cf4916ec2c87c7c1d83, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/fd4478ca22164caa9eecaa98a945d3d0, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/0c8e6efed97a44658ef7188ab811de74, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/ac33a56c447f4806bab626b20fdcf70f, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/2e46722a70684cecb1d46eccdb40343d, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/89a20537a1824ef7b5c4fd5cc79e351a, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/3b95b18cd7194c839aa240eae38ed958] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp, totalSize=81.3 K 2023-04-20 12:56:13,585 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 5d85ed4b5a7b4cf4916ec2c87c7c1d83, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=22, earliestPutTs=1681995366895 2023-04-20 12:56:13,585 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting fd4478ca22164caa9eecaa98a945d3d0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=38, earliestPutTs=1681995366899 2023-04-20 12:56:13,586 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 0c8e6efed97a44658ef7188ab811de74, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=54, earliestPutTs=1681995369031 2023-04-20 12:56:13,586 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting ac33a56c447f4806bab626b20fdcf70f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1681995369652 2023-04-20 12:56:13,586 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 2e46722a70684cecb1d46eccdb40343d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1681995370770 2023-04-20 12:56:13,586 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 89a20537a1824ef7b5c4fd5cc79e351a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=104, earliestPutTs=1681995371918 2023-04-20 12:56:13,587 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 3b95b18cd7194c839aa240eae38ed958, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=119, earliestPutTs=1681995372539 2023-04-20 12:56:13,597 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): 8113dc3e39e3420c86d60a8170c364f8#C#compaction#505 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:56:13,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 8113dc3e39e3420c86d60a8170c364f8 2023-04-20 12:56:13,783 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:13,795 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:13,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41978 deadline: 1681995433792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,795 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:13,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42008 deadline: 1681995433792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,796 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:13,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42076 deadline: 1681995433793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,796 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:13,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42018 deadline: 1681995433793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,796 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:13,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42016 deadline: 1681995433793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,898 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:13,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41978 deadline: 1681995433896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,898 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:13,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42008 deadline: 1681995433896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,898 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:13,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42076 deadline: 1681995433896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,898 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:13,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42016 deadline: 1681995433897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,899 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:13,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42018 deadline: 1681995433897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:13,989 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420d163f7bbb3084794806ee281cd4ff788 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420d163f7bbb3084794806ee281cd4ff788 2023-04-20 12:56:14,012 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/C/8041c0eb51e249009ed8d3e439bbc2b2 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/8041c0eb51e249009ed8d3e439bbc2b2 2023-04-20 12:56:14,016 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 8113dc3e39e3420c86d60a8170c364f8/C of 8113dc3e39e3420c86d60a8170c364f8 into 8041c0eb51e249009ed8d3e439bbc2b2(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:56:14,016 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8113dc3e39e3420c86d60a8170c364f8: 2023-04-20 12:56:14,016 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8., storeName=8113dc3e39e3420c86d60a8170c364f8/C, priority=9, startTime=1681995373502; duration=0sec 2023-04-20 12:56:14,016 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:56:14,102 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:14,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42016 deadline: 1681995434099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:14,102 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:14,102 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:14,102 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:14,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42008 deadline: 1681995434100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:14,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42076 deadline: 1681995434099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:14,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41978 deadline: 1681995434100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:14,103 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:14,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42018 deadline: 1681995434100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:14,393 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=136, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/A/ab12c5e34e064f1c8504d5fe878add97 2023-04-20 12:56:14,403 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=136 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/B/d06e6a61e9444b7daf40d0040496eb54 2023-04-20 12:56:14,406 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:14,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42076 deadline: 1681995434403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:14,406 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:14,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42018 deadline: 1681995434404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:14,407 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:14,407 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:14,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42008 deadline: 1681995434404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:14,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41978 deadline: 1681995434404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:14,407 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:14,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42016 deadline: 1681995434405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:14,416 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=136 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/C/9990be75bc4241e0adbfbe1feedf6323 2023-04-20 12:56:14,418 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/A/ef945febbc434b91ba3d74223e35c169 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/ef945febbc434b91ba3d74223e35c169 2023-04-20 12:56:14,420 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/A/ab12c5e34e064f1c8504d5fe878add97 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/ab12c5e34e064f1c8504d5fe878add97 2023-04-20 12:56:14,423 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 8113dc3e39e3420c86d60a8170c364f8/A of 8113dc3e39e3420c86d60a8170c364f8 into ef945febbc434b91ba3d74223e35c169(size=25.4 K), total size for store is 25.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:56:14,423 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8113dc3e39e3420c86d60a8170c364f8: 2023-04-20 12:56:14,423 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8., storeName=8113dc3e39e3420c86d60a8170c364f8/A, priority=9, startTime=1681995373502; duration=0sec 2023-04-20 12:56:14,423 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:56:14,425 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/ab12c5e34e064f1c8504d5fe878add97, entries=150, sequenceid=136, filesize=25.3 K 2023-04-20 12:56:14,426 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/B/d06e6a61e9444b7daf40d0040496eb54 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/d06e6a61e9444b7daf40d0040496eb54 2023-04-20 12:56:14,429 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/d06e6a61e9444b7daf40d0040496eb54, entries=150, sequenceid=136, filesize=11.8 K 2023-04-20 12:56:14,430 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/C/9990be75bc4241e0adbfbe1feedf6323 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/9990be75bc4241e0adbfbe1feedf6323 2023-04-20 12:56:14,433 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/9990be75bc4241e0adbfbe1feedf6323, entries=150, sequenceid=136, filesize=11.8 K 2023-04-20 12:56:14,434 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for 8113dc3e39e3420c86d60a8170c364f8 in 916ms, sequenceid=136, compaction requested=false 2023-04-20 12:56:14,434 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8113dc3e39e3420c86d60a8170c364f8: 2023-04-20 12:56:14,434 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:14,434 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:14,434 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:14,434 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:14,434 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:14,434 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:14,436 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:14,436 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:14,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:14,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:14,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:14,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:14,437 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:14,437 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:14,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:14,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:14,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:14,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:14,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:14,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24f3fb13[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:14,438 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:14,438 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:14,439 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:14,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:14,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:14,439 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:14,439 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:14,439 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:14,439 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:14,439 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:14,440 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:14,440 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:14,440 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:14,440 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:14,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:14,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:14,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:14,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:14,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:14,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:14,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:14,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:14,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:14,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:14,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:14,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:14,442 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:14,442 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:14,442 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:14,442 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:14,442 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:14,443 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:14,443 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:14,443 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:14,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:14,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:14,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:14,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:14,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:14,443 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:14,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:14,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:14,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:14,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:14,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:14,444 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:14,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:14,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:14,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:14,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:14,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:14,445 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:14,445 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:14,445 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:14,445 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:14,445 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:14,445 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:14,445 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:14,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:14,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:14,446 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:14,446 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:14,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:14,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:14,446 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:14,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:14,446 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:14,446 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:14,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:14,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:14,546 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:14,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:14,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:14,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:14,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:14,548 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:14,548 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:14,548 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:14,549 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:14,549 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:14,549 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:14,549 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:14,549 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:14,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:14,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:14,550 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:14,550 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:14,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:14,550 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:14,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:14,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:14,550 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:14,551 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:14,551 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:14,551 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:14,551 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:14,551 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:14,551 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:14,551 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:14,551 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:14,551 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:14,551 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8113dc3e39e3420c86d60a8170c364f8 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-20 12:56:14,551 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=A 2023-04-20 12:56:14,551 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:14,551 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=B 2023-04-20 12:56:14,551 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:14,551 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=C 2023-04-20 12:56:14,551 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:14,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 8113dc3e39e3420c86d60a8170c364f8 2023-04-20 12:56:14,929 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:14,943 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:14,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41978 deadline: 1681995434939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:14,944 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:14,944 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:14,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42016 deadline: 1681995434939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:14,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42076 deadline: 1681995434939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:14,944 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:14,944 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:14,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42018 deadline: 1681995434940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:14,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42008 deadline: 1681995434940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:14,960 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023042039efa8df50834757b0d07003ad8ca949 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042039efa8df50834757b0d07003ad8ca949 2023-04-20 12:56:15,045 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:15,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41978 deadline: 1681995435044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:15,046 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:15,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42016 deadline: 1681995435045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:15,046 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:15,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42076 deadline: 1681995435045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:15,046 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:15,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42018 deadline: 1681995435045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:15,046 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:15,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42008 deadline: 1681995435045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:15,249 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:15,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42016 deadline: 1681995435247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:15,249 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:15,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42076 deadline: 1681995435247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:15,249 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:15,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42018 deadline: 1681995435247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:15,250 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:15,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41978 deadline: 1681995435247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:15,250 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:15,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42008 deadline: 1681995435247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:15,364 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=155, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/A/475193737fc24ed789fc466efe4b52bd 2023-04-20 12:56:15,377 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=155 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/B/015c558d946e464189962e779e9fa3a6 2023-04-20 12:56:15,552 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:15,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42008 deadline: 1681995435551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:15,552 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:15,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42018 deadline: 1681995435551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:15,553 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:15,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42016 deadline: 1681995435552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:15,553 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:15,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42076 deadline: 1681995435552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:15,554 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:15,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41978 deadline: 1681995435552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:15,784 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=155 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/C/6bd9cd0da872410fb37968229f701cc8 2023-04-20 12:56:15,787 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/A/475193737fc24ed789fc466efe4b52bd as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/475193737fc24ed789fc466efe4b52bd 2023-04-20 12:56:15,790 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/475193737fc24ed789fc466efe4b52bd, entries=150, sequenceid=155, filesize=25.3 K 2023-04-20 12:56:15,790 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/B/015c558d946e464189962e779e9fa3a6 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/015c558d946e464189962e779e9fa3a6 2023-04-20 12:56:15,793 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/015c558d946e464189962e779e9fa3a6, entries=150, sequenceid=155, filesize=11.8 K 2023-04-20 12:56:15,793 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/C/6bd9cd0da872410fb37968229f701cc8 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/6bd9cd0da872410fb37968229f701cc8 2023-04-20 12:56:15,796 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/6bd9cd0da872410fb37968229f701cc8, entries=150, sequenceid=155, filesize=11.8 K 2023-04-20 12:56:15,796 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 8113dc3e39e3420c86d60a8170c364f8 in 1245ms, sequenceid=155, compaction requested=true 2023-04-20 12:56:15,796 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8113dc3e39e3420c86d60a8170c364f8: 2023-04-20 12:56:15,796 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:15,797 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:15,797 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:15,797 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:15,797 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:15,797 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:15,797 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:15,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:15,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:15,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:15,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:15,798 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:15,798 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:15,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:15,798 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:15,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:15,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:15,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:15,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:15,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e40d1bf[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:15,799 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:15,799 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:15,799 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:15,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:15,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:15,800 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:15,800 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:15,800 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:15,800 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:15,800 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:15,800 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:15,800 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:15,800 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:15,800 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:15,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:15,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:15,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:15,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:15,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:15,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:15,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:15,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:15,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:15,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:15,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:15,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:15,802 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:15,802 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:15,803 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:15,803 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:15,803 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:15,803 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:15,803 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:15,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:15,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:15,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:15,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:15,803 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:15,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:15,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:15,804 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:15,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:15,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:15,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:15,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:15,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:15,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:15,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:15,805 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:15,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:15,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:15,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:15,806 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:15,806 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:15,806 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:15,806 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:15,806 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:15,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:15,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:15,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:15,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:15,807 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:15,807 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:15,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:15,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:15,807 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:15,807 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:15,807 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:15,807 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:15,807 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:15,907 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:15,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:15,909 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:15,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:15,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:15,910 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:15,910 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:15,910 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:15,910 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:15,910 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:15,910 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:15,911 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:15,911 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:15,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:15,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:15,911 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:15,911 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:15,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:15,912 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:15,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:15,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:15,912 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:15,912 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:15,912 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:15,912 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:15,912 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:15,912 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:15,912 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:15,912 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:15,912 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:15,912 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:15,912 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8113dc3e39e3420c86d60a8170c364f8 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-20 12:56:15,913 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=A 2023-04-20 12:56:15,913 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:15,913 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=B 2023-04-20 12:56:15,913 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:15,913 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=C 2023-04-20 12:56:15,913 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:15,920 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420fc27da1762984c2fa411ca67892582ce to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420fc27da1762984c2fa411ca67892582ce 2023-04-20 12:56:16,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 8113dc3e39e3420c86d60a8170c364f8 2023-04-20 12:56:16,055 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:16,065 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:16,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42076 deadline: 1681995436061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:16,065 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:16,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42016 deadline: 1681995436063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:16,067 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:16,067 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:16,067 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:16,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41978 deadline: 1681995436065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:16,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42018 deadline: 1681995436064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:16,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42008 deadline: 1681995436064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:16,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:16,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42076 deadline: 1681995436165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:16,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:16,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42016 deadline: 1681995436165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:16,169 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:16,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41978 deadline: 1681995436168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:16,169 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:16,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42018 deadline: 1681995436168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:16,169 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:16,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42008 deadline: 1681995436168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:16,325 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=171, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/A/52f3560285f044ceb2059bf69522f0c9 2023-04-20 12:56:16,369 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:16,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42016 deadline: 1681995436368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:16,369 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:16,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42076 deadline: 1681995436368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:16,371 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:16,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42018 deadline: 1681995436370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:16,371 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:16,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42008 deadline: 1681995436370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:16,372 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:16,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41978 deadline: 1681995436371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:16,671 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:16,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42016 deadline: 1681995436671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:16,672 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:16,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42076 deadline: 1681995436671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:16,674 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:16,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42018 deadline: 1681995436672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:16,674 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:16,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42008 deadline: 1681995436673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:16,674 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:16,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41978 deadline: 1681995436673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:16,739 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=171 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/B/cbc4d02c68ed4708ae7e7e09ff150dc8 2023-04-20 12:56:16,750 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=171 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/C/2f4a9d359c4d429091ad9bf8373a4ff1 2023-04-20 12:56:16,753 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/A/52f3560285f044ceb2059bf69522f0c9 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/52f3560285f044ceb2059bf69522f0c9 2023-04-20 12:56:16,756 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/52f3560285f044ceb2059bf69522f0c9, entries=150, sequenceid=171, filesize=25.3 K 2023-04-20 12:56:16,757 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/B/cbc4d02c68ed4708ae7e7e09ff150dc8 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/cbc4d02c68ed4708ae7e7e09ff150dc8 2023-04-20 12:56:16,760 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/cbc4d02c68ed4708ae7e7e09ff150dc8, entries=150, sequenceid=171, filesize=11.8 K 2023-04-20 12:56:16,760 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/C/2f4a9d359c4d429091ad9bf8373a4ff1 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/2f4a9d359c4d429091ad9bf8373a4ff1 2023-04-20 12:56:16,763 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/2f4a9d359c4d429091ad9bf8373a4ff1, entries=150, sequenceid=171, filesize=11.8 K 2023-04-20 12:56:16,763 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 8113dc3e39e3420c86d60a8170c364f8 in 851ms, sequenceid=171, compaction requested=true 2023-04-20 12:56:16,763 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8113dc3e39e3420c86d60a8170c364f8: 2023-04-20 12:56:16,763 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:16,763 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:16,763 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:16,763 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:16,763 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:16,763 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:16,764 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:16,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:16,764 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:16,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:16,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:16,765 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:16,765 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:16,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:16,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:16,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:16,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:16,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:16,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:16,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ef2b991[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:16,766 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:16,766 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:16,766 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:16,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:16,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:16,766 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:16,766 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:16,766 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:16,766 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:16,766 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:16,767 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:16,767 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:16,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:16,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:16,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:16,767 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:16,767 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:16,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:16,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:16,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:16,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:16,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:16,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:16,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:16,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:16,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:16,769 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:16,769 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:16,769 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:16,769 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:16,769 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:16,770 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:16,770 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:16,770 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:16,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:16,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:16,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:16,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:16,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:16,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:16,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:16,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:16,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:16,771 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:16,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:16,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:16,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:16,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:16,772 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:16,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:16,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:16,773 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:16,773 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:16,773 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:16,773 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:16,773 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:16,773 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:16,773 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:16,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:16,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:16,773 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:16,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:16,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:16,774 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:16,774 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:16,773 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:16,774 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:16,774 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:16,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:16,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:16,874 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:16,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:16,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:16,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:16,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:16,876 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:16,876 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:16,876 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:16,877 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:16,877 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:16,877 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:16,877 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:16,877 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:16,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:16,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:16,878 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:16,878 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:16,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:16,878 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:16,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:16,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:16,878 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:16,878 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:16,878 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:16,878 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:16,878 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:16,878 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:16,879 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:16,879 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:16,879 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:16,879 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:16,879 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8113dc3e39e3420c86d60a8170c364f8 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-20 12:56:16,879 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=A 2023-04-20 12:56:16,879 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:16,879 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=B 2023-04-20 12:56:16,879 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:16,879 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=C 2023-04-20 12:56:16,879 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:17,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 8113dc3e39e3420c86d60a8170c364f8 2023-04-20 12:56:17,175 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:17,187 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:17,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42076 deadline: 1681995437185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:17,189 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:17,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41978 deadline: 1681995437186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:17,189 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:17,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42016 deadline: 1681995437186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:17,189 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:17,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42018 deadline: 1681995437187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:17,190 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:17,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42008 deadline: 1681995437187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:17,290 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:17,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42076 deadline: 1681995437288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:17,290 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420c217beb2ead14193971976ce00af00d5 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420c217beb2ead14193971976ce00af00d5 2023-04-20 12:56:17,292 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:17,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42016 deadline: 1681995437290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:17,292 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:17,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42008 deadline: 1681995437290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:17,292 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:17,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42018 deadline: 1681995437291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:17,292 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:17,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41978 deadline: 1681995437291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:17,295 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=187, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/A/f6c0276a42f14a699bd6cbe62d202a85 2023-04-20 12:56:17,302 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=187 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/B/e9aa6376c3de4fd6b747f739fa0b5dfc 2023-04-20 12:56:17,309 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=187 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/C/9913a77d6b1d424eb22bc682cb7290f7 2023-04-20 12:56:17,312 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/A/f6c0276a42f14a699bd6cbe62d202a85 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/f6c0276a42f14a699bd6cbe62d202a85 2023-04-20 12:56:17,315 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/f6c0276a42f14a699bd6cbe62d202a85, entries=150, sequenceid=187, filesize=25.3 K 2023-04-20 12:56:17,316 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/B/e9aa6376c3de4fd6b747f739fa0b5dfc as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/e9aa6376c3de4fd6b747f739fa0b5dfc 2023-04-20 12:56:17,319 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/e9aa6376c3de4fd6b747f739fa0b5dfc, entries=150, sequenceid=187, filesize=11.8 K 2023-04-20 12:56:17,319 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/C/9913a77d6b1d424eb22bc682cb7290f7 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/9913a77d6b1d424eb22bc682cb7290f7 2023-04-20 12:56:17,322 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/9913a77d6b1d424eb22bc682cb7290f7, entries=150, sequenceid=187, filesize=11.8 K 2023-04-20 12:56:17,322 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for 8113dc3e39e3420c86d60a8170c364f8 in 443ms, sequenceid=187, compaction requested=true 2023-04-20 12:56:17,322 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8113dc3e39e3420c86d60a8170c364f8: 2023-04-20 12:56:17,322 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:17,322 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:17,323 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:17,323 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:17,323 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:17,323 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:17,323 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:17,323 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:17,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:17,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:17,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:17,324 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:17,324 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:17,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:17,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:17,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:17,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:17,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:17,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:17,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a097057[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:17,325 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:17,325 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:17,326 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:17,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:17,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:17,326 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:17,326 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:17,326 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:17,327 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:17,327 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:17,327 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:17,327 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:17,327 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:17,327 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:17,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:17,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:17,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:17,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:17,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:17,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:17,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:17,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:17,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:17,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:17,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:17,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:17,329 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:17,329 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:17,329 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:17,329 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:17,329 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:17,330 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:17,330 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:17,330 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:17,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:17,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:17,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:17,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:17,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:17,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:17,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:17,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:17,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:17,331 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:17,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:17,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:17,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:17,331 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:17,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:17,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:17,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:17,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:17,333 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:17,333 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:17,333 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:17,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:17,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:17,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:17,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:17,333 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:17,333 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:17,334 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:17,334 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:17,334 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:17,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:17,334 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:17,334 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:17,334 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:17,334 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:17,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:17,334 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:17,434 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:17,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:17,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:17,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:17,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:17,437 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:17,437 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:17,437 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:17,437 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:17,437 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:17,437 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:17,438 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:17,438 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:17,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:17,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:17,438 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:17,438 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:17,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:17,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:17,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:17,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:17,439 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:17,439 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:17,439 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:17,439 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:17,439 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:17,439 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:17,439 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:17,439 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:17,439 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:17,439 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:17,439 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8113dc3e39e3420c86d60a8170c364f8 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-20 12:56:17,439 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=A 2023-04-20 12:56:17,440 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:17,440 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=B 2023-04-20 12:56:17,440 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:17,440 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=C 2023-04-20 12:56:17,440 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:17,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 8113dc3e39e3420c86d60a8170c364f8 2023-04-20 12:56:17,495 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:17,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:17,506 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:17,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41978 deadline: 1681995437502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:17,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42016 deadline: 1681995437502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:17,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:17,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42018 deadline: 1681995437503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:17,507 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:17,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42076 deadline: 1681995437504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:17,508 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:17,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42008 deadline: 1681995437506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:17,610 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:17,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41978 deadline: 1681995437607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:17,610 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:17,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42018 deadline: 1681995437607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:17,610 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:17,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42016 deadline: 1681995437607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:17,610 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:17,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42076 deadline: 1681995437607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:17,610 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:17,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42008 deadline: 1681995437609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:17,812 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:17,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41978 deadline: 1681995437810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:17,812 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:17,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42018 deadline: 1681995437811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:17,813 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:17,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42016 deadline: 1681995437812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:17,813 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:17,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42076 deadline: 1681995437812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:17,814 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:17,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42008 deadline: 1681995437812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:17,850 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420d422bc54fcde4d8891a55ad80473aaf0 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420d422bc54fcde4d8891a55ad80473aaf0 2023-04-20 12:56:18,116 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:18,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42016 deadline: 1681995438114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:18,117 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:18,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41978 deadline: 1681995438115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:18,117 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:18,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42018 deadline: 1681995438115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:18,117 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:18,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42008 deadline: 1681995438115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:18,118 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:18,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42076 deadline: 1681995438116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:18,254 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=204, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/A/ed61dc61ec2e4471a8dfd8fadaf8c35e 2023-04-20 12:56:18,620 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:18,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42076 deadline: 1681995438618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:18,620 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:18,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42018 deadline: 1681995438619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:18,620 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:18,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42008 deadline: 1681995438619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:18,623 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:18,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42016 deadline: 1681995438621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:18,623 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:18,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41978 deadline: 1681995438621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:18,662 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=204 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/B/88803527119348b4ada46bd7f6e7cd18 2023-04-20 12:56:18,669 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=204 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/C/c364e0615c134c7a8e30f977d2eee77a 2023-04-20 12:56:18,673 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/A/ed61dc61ec2e4471a8dfd8fadaf8c35e as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/ed61dc61ec2e4471a8dfd8fadaf8c35e 2023-04-20 12:56:18,676 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/ed61dc61ec2e4471a8dfd8fadaf8c35e, entries=150, sequenceid=204, filesize=25.3 K 2023-04-20 12:56:18,677 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/B/88803527119348b4ada46bd7f6e7cd18 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/88803527119348b4ada46bd7f6e7cd18 2023-04-20 12:56:18,680 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/88803527119348b4ada46bd7f6e7cd18, entries=150, sequenceid=204, filesize=11.8 K 2023-04-20 12:56:18,680 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/C/c364e0615c134c7a8e30f977d2eee77a as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/c364e0615c134c7a8e30f977d2eee77a 2023-04-20 12:56:18,684 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/c364e0615c134c7a8e30f977d2eee77a, entries=150, sequenceid=204, filesize=11.8 K 2023-04-20 12:56:18,685 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 8113dc3e39e3420c86d60a8170c364f8 in 1246ms, sequenceid=204, compaction requested=true 2023-04-20 12:56:18,685 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8113dc3e39e3420c86d60a8170c364f8: 2023-04-20 12:56:18,686 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:18,686 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:18,686 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:18,686 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:18,686 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:18,686 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:18,686 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:18,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:18,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:18,687 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:18,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:18,687 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:18,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:18,687 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:18,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:18,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:18,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:18,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:18,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:18,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21992b66[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:18,688 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:18,688 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:18,689 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:18,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:18,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:18,689 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:18,689 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:18,689 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:18,689 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:18,689 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:18,690 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:18,690 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:18,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:18,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:18,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:18,690 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:18,690 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:18,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:18,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:18,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:18,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:18,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:18,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:18,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:18,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:18,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:18,692 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:18,692 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:18,692 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:18,692 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:18,692 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:18,693 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:18,693 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:18,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:18,693 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:18,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:18,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:18,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:18,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:18,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:18,694 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:18,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:18,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:18,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:18,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:18,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:18,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:18,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:18,695 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:18,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:18,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:18,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:18,696 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:18,696 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:18,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:18,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:18,697 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:18,697 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:18,697 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:18,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:18,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:18,697 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:18,697 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:18,697 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:18,697 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:18,697 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:18,697 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:18,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:18,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:18,697 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:18,697 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:18,797 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:18,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:18,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:18,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:18,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:18,800 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:18,800 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:18,800 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:18,800 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:18,800 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:18,800 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:18,801 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:18,801 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:18,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:18,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:18,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:18,801 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:18,801 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:18,802 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:18,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:18,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:18,802 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:18,802 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:18,802 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:18,802 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:18,802 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:18,802 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:18,802 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:18,802 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:18,802 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:18,802 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:18,803 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8113dc3e39e3420c86d60a8170c364f8 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-20 12:56:18,803 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=A 2023-04-20 12:56:18,803 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:18,803 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=B 2023-04-20 12:56:18,803 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:18,803 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=C 2023-04-20 12:56:18,803 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:18,812 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304201e06d8a22ce64c8c86b877089259001b to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304201e06d8a22ce64c8c86b877089259001b 2023-04-20 12:56:18,816 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=219, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/A/56f748456cc046f1b054c6753401e26f 2023-04-20 12:56:19,223 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=219 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/B/3578e057e5e34551a249bf5d9f59fc6d 2023-04-20 12:56:19,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 8113dc3e39e3420c86d60a8170c364f8 2023-04-20 12:56:19,624 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:19,630 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=219 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/C/13e1c545f4db47329cb58643a62e5f81 2023-04-20 12:56:19,634 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/A/56f748456cc046f1b054c6753401e26f as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/56f748456cc046f1b054c6753401e26f 2023-04-20 12:56:19,638 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/56f748456cc046f1b054c6753401e26f, entries=150, sequenceid=219, filesize=25.3 K 2023-04-20 12:56:19,638 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/B/3578e057e5e34551a249bf5d9f59fc6d as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/3578e057e5e34551a249bf5d9f59fc6d 2023-04-20 12:56:19,641 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/3578e057e5e34551a249bf5d9f59fc6d, entries=150, sequenceid=219, filesize=11.8 K 2023-04-20 12:56:19,642 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/C/13e1c545f4db47329cb58643a62e5f81 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/13e1c545f4db47329cb58643a62e5f81 2023-04-20 12:56:19,644 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/13e1c545f4db47329cb58643a62e5f81, entries=150, sequenceid=219, filesize=11.8 K 2023-04-20 12:56:19,645 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for 8113dc3e39e3420c86d60a8170c364f8 in 842ms, sequenceid=219, compaction requested=true 2023-04-20 12:56:19,645 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8113dc3e39e3420c86d60a8170c364f8: 2023-04-20 12:56:19,645 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:19,645 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:19,645 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:19,645 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,645 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:19,645 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:19,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 8113dc3e39e3420c86d60a8170c364f8 2023-04-20 12:56:19,647 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8113dc3e39e3420c86d60a8170c364f8 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-20 12:56:19,647 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=A 2023-04-20 12:56:19,647 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:19,647 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=B 2023-04-20 12:56:19,647 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:19,648 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=C 2023-04-20 12:56:19,648 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:19,648 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,648 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:19,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:19,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:19,648 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:19,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:19,648 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:19,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:19,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:19,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:19,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:19,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@394ccbc3[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:19,650 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:19,650 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:19,650 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:19,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:19,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:19,650 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:19,650 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:19,650 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:19,650 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,650 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:19,651 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:19,651 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,651 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:19,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:19,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:19,651 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:19,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:19,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:19,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:19,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:19,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:19,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:19,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:19,653 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:19,653 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:19,653 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:19,653 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:19,653 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:19,654 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:19,654 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:19,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:19,654 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:19,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:19,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:19,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:19,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:19,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:19,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:19,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:19,655 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:19,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:19,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:19,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:19,656 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:19,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:19,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,657 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,657 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:19,657 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:19,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:19,657 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:19,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:19,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:19,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:19,657 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:19,657 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:19,658 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:19,658 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:19,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:19,658 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:19,658 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:19,658 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:19,658 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:19,658 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:19,658 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304209670b44cccf648d58a2ab9baf5dcc48e to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304209670b44cccf648d58a2ab9baf5dcc48e 2023-04-20 12:56:19,660 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:19,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42018 deadline: 1681995439654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,660 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:19,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42008 deadline: 1681995439655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,661 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:19,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41978 deadline: 1681995439656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,661 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:19,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42076 deadline: 1681995439656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,663 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:19,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42016 deadline: 1681995439660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,758 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:19,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:19,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:19,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:19,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:19,761 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:19,761 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:19,761 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:19,761 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:19,761 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:19,761 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:19,762 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:19,762 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:19,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:19,762 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:19,762 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42018 deadline: 1681995439761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,762 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:19,762 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:19,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42008 deadline: 1681995439761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:19,762 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:19,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41978 deadline: 1681995439762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:19,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:19,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:19,763 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:19,763 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:19,763 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:19,763 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:19,763 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:19,763 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:19,763 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:19,763 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:19,763 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:19,763 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:19,764 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:19,764 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:19,764 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:19,764 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:19,764 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,764 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:19,764 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:19,764 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,764 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:19,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:19,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:19,765 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:19,765 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:19,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:19,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:19,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:19,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,766 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:19,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42076 deadline: 1681995439762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:19,766 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:19,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42016 deadline: 1681995439764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:19,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7239c34a[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:19,766 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:19,766 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:19,767 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:19,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:19,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:19,767 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:19,767 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,767 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:19,767 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:19,767 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:19,767 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:19,767 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:19,767 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,767 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:19,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:19,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:19,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:19,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:19,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:19,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:19,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:19,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:19,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:19,769 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:19,769 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:19,770 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:19,770 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:19,770 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:19,771 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:19,771 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:19,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:19,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:19,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:19,772 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:19,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:19,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:19,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:19,772 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:19,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:19,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:19,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:19,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:19,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:19,773 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:19,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:19,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,774 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,774 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:19,774 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:19,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:19,774 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:19,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:19,774 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:19,774 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:19,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:19,774 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:19,774 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:19,774 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:19,774 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:19,775 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:19,775 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:19,775 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:19,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:19,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:19,875 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:19,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:19,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:19,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:19,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:19,877 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:19,877 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:19,877 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:19,877 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:19,878 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:19,878 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:19,878 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,878 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:19,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:19,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:19,879 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,879 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:19,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:19,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:19,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:19,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:19,879 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:19,879 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:19,879 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:19,879 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:19,879 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:19,879 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:19,880 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:19,880 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:19,880 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:19,880 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:19,880 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:19,880 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:19,880 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:19,880 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:19,880 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,880 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:19,880 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:19,881 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,881 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:19,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:19,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:19,881 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:19,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:19,881 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:19,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:19,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:19,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:19,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:19,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23638e92[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:19,882 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:19,882 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:19,882 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:19,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:19,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:19,883 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:19,883 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:19,883 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:19,883 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,883 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:19,883 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,883 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:19,883 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:19,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:19,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:19,883 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:19,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:19,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:19,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:19,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:19,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:19,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:19,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:19,885 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:19,885 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:19,885 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:19,885 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:19,885 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:19,886 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:19,886 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:19,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:19,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:19,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:19,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:19,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:19,886 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:19,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:19,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:19,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:19,887 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:19,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:19,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:19,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:19,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,887 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:19,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:19,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,889 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,889 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:19,889 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:19,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:19,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:19,889 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:19,889 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,889 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:19,889 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:19,889 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:19,889 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:19,889 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:19,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:19,889 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:19,889 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:19,889 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:19,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:19,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:19,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:19,964 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:19,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42018 deadline: 1681995439963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,964 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:19,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42008 deadline: 1681995439963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,965 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:19,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41978 deadline: 1681995439963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,968 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:19,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42076 deadline: 1681995439966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,969 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:19,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42016 deadline: 1681995439968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,989 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:19,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:19,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:19,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:19,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:19,992 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:19,992 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:19,992 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:19,992 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:19,992 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:19,993 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:19,993 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:19,993 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:19,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:19,993 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,993 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:19,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:19,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:19,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:19,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:19,994 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:19,994 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:19,994 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:19,994 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:19,994 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:19,994 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:19,994 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:19,995 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:19,995 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:19,995 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:19,995 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:19,995 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:19,995 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:19,995 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:19,995 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,995 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:19,995 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:19,996 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,996 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:19,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:19,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:19,996 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:19,996 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:19,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:19,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:19,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:19,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:19,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:19,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b494383[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:19,997 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:19,997 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:19,998 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:19,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:19,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:19,998 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:19,998 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:19,998 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,998 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:19,998 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:19,998 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:19,998 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:19,998 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,999 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:19,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:19,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:19,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:19,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:19,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:19,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:20,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:20,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:20,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:20,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:20,001 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:20,001 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:20,001 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:20,001 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:20,001 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:20,001 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:20,001 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:20,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:20,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:20,001 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:20,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:20,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:20,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:20,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:20,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:20,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:20,003 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:20,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:20,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:20,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:20,004 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:20,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:20,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,005 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:20,005 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:20,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:20,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:20,005 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,005 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:20,005 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:20,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:20,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:20,005 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:20,005 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,006 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:20,006 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:20,006 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:20,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:20,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:20,006 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:20,006 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:20,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:20,062 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=239, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/A/25a218c148c6407b9b153833c1f692c1 2023-04-20 12:56:20,106 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:20,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:20,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:20,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:20,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:20,109 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:20,109 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:20,109 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:20,109 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:20,109 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:20,109 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:20,110 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:20,110 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:20,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:20,110 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,110 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:20,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:20,110 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:20,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:20,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:20,111 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:20,111 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:20,111 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:20,111 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:20,111 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:20,111 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:20,111 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:20,111 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:20,111 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:20,111 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:20,111 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:20,111 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:20,111 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:20,111 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:20,111 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,111 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:20,111 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:20,112 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,112 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:20,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:20,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:20,112 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:20,112 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:20,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:20,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:20,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:20,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:20,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:20,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14048bfa[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:20,114 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:20,114 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:20,114 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:20,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:20,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:20,114 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:20,114 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:20,114 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:20,114 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,114 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:20,115 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,115 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:20,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:20,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:20,115 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:20,115 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:20,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:20,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:20,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:20,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:20,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:20,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:20,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:20,117 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:20,117 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:20,117 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:20,117 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:20,117 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:20,118 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:20,118 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:20,118 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:20,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:20,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:20,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:20,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:20,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:20,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:20,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:20,118 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:20,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:20,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:20,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:20,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:20,119 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:20,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:20,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,121 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,122 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:20,122 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:20,122 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:20,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:20,122 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:20,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:20,122 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:20,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:20,122 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:20,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:20,122 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:20,122 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:20,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:20,122 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:20,122 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:20,122 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:20,122 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:20,222 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:20,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:20,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:20,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:20,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:20,225 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:20,225 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:20,225 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:20,225 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:20,225 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:20,225 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:20,226 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:20,226 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:20,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:20,226 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:20,226 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:20,227 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:20,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:20,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:20,227 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:20,227 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:20,227 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:20,227 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:20,227 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:20,227 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:20,227 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:20,227 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:20,227 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:20,227 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:20,228 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:20,228 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:20,228 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:20,228 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:20,228 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,228 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:20,228 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:20,228 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:20,228 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:20,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:20,229 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:20,229 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:20,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:20,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:20,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:20,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:20,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:20,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@289b39a3[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:20,230 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:20,230 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:20,231 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:20,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:20,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:20,231 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:20,231 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:20,231 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:20,231 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,231 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:20,232 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,232 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:20,232 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:20,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:20,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:20,232 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:20,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:20,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:20,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:20,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:20,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:20,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:20,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:20,234 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:20,234 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:20,234 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:20,234 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:20,234 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:20,235 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:20,235 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:20,235 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:20,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:20,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:20,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:20,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:20,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:20,235 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:20,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:20,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:20,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:20,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:20,236 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:20,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:20,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:20,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:20,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,238 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:20,238 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,238 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:20,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:20,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:20,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:20,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:20,238 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:20,238 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:20,238 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:20,238 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:20,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:20,238 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:20,238 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:20,238 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:20,238 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:20,239 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:20,239 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:20,267 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:20,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41978 deadline: 1681995440265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,267 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:20,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42018 deadline: 1681995440266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,268 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:20,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42008 deadline: 1681995440266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,272 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:20,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42076 deadline: 1681995440270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,273 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:20,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42016 deadline: 1681995440270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,339 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:20,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:20,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:20,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:20,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:20,341 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:20,341 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:20,341 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:20,341 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:20,342 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:20,342 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:20,342 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:20,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:20,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:20,343 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:20,343 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,343 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:20,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:20,343 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:20,344 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:20,344 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:20,344 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:20,344 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:20,344 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:20,344 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:20,344 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:20,344 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:20,344 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:20,344 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:20,344 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:20,344 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:20,344 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:20,344 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,344 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:20,344 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:20,345 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,345 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:20,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:20,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:20,345 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:20,345 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:20,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:20,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:20,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:20,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:20,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:20,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1733253c[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:20,346 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:20,346 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:20,347 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:20,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:20,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:20,347 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:20,347 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:20,347 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:20,347 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,347 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:20,348 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,348 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:20,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:20,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:20,348 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:20,348 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:20,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:20,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:20,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:20,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:20,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:20,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:20,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:20,350 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:20,350 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:20,350 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:20,350 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:20,350 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:20,351 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:20,351 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:20,351 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:20,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:20,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:20,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:20,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:20,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:20,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:20,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:20,351 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:20,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:20,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:20,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:20,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:20,352 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:20,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:20,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,353 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:20,353 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:20,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:20,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:20,353 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:20,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:20,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:20,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:20,353 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,354 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:20,354 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:20,354 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:20,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:20,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:20,354 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:20,354 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,354 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:20,354 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:20,354 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:20,454 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:20,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:20,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:20,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:20,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:20,457 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:20,457 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:20,457 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:20,457 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:20,457 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:20,457 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:20,458 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:20,458 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:20,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:20,458 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,459 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:20,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:20,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:20,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:20,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:20,459 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:20,459 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:20,459 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:20,459 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:20,459 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:20,459 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:20,460 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:20,460 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:20,460 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:20,460 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:20,460 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:20,460 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:20,460 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:20,460 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:20,460 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,460 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:20,460 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:20,461 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,461 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:20,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:20,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:20,461 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:20,461 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:20,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:20,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:20,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:20,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:20,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:20,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16672902[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:20,462 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:20,462 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:20,463 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:20,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:20,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:20,463 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,463 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:20,463 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:20,463 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:20,463 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:20,464 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,464 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:20,464 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:20,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:20,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:20,464 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:20,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:20,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:20,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:20,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:20,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:20,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:20,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:20,465 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:20,465 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:20,466 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:20,466 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:20,466 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:20,466 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:20,466 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:20,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:20,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:20,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:20,466 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:20,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:20,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:20,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:20,467 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:20,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:20,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:20,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:20,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:20,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:20,468 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:20,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:20,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,469 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,470 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:20,470 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:20,470 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:20,470 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,470 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:20,470 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:20,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:20,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:20,470 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:20,470 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:20,470 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:20,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:20,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:20,470 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=239 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/B/a318c92e343d48f9b09a4f791653b6c6 2023-04-20 12:56:20,470 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:20,470 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:20,470 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:20,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:20,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:20,477 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=239 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/C/0a9c01fcbaf640f1b129e139d6cc7268 2023-04-20 12:56:20,480 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/A/25a218c148c6407b9b153833c1f692c1 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/25a218c148c6407b9b153833c1f692c1 2023-04-20 12:56:20,482 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/25a218c148c6407b9b153833c1f692c1, entries=250, sequenceid=239, filesize=38.9 K 2023-04-20 12:56:20,483 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/B/a318c92e343d48f9b09a4f791653b6c6 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/a318c92e343d48f9b09a4f791653b6c6 2023-04-20 12:56:20,486 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/a318c92e343d48f9b09a4f791653b6c6, entries=150, sequenceid=239, filesize=11.8 K 2023-04-20 12:56:20,486 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/C/0a9c01fcbaf640f1b129e139d6cc7268 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/0a9c01fcbaf640f1b129e139d6cc7268 2023-04-20 12:56:20,489 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/0a9c01fcbaf640f1b129e139d6cc7268, entries=150, sequenceid=239, filesize=11.8 K 2023-04-20 12:56:20,490 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 8113dc3e39e3420c86d60a8170c364f8 in 843ms, sequenceid=239, compaction requested=true 2023-04-20 12:56:20,490 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8113dc3e39e3420c86d60a8170c364f8: 2023-04-20 12:56:20,490 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:56:20,490 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-20 12:56:20,490 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-20 12:56:20,490 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:56:20,490 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:56:20,493 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 221242 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-20 12:56:20,493 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 96378 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-20 12:56:20,493 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): 8113dc3e39e3420c86d60a8170c364f8/A is initiating minor compaction (all files) 2023-04-20 12:56:20,493 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): 8113dc3e39e3420c86d60a8170c364f8/B is initiating minor compaction (all files) 2023-04-20 12:56:20,493 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8113dc3e39e3420c86d60a8170c364f8/A in TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:20,493 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8113dc3e39e3420c86d60a8170c364f8/B in TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:20,493 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/ef945febbc434b91ba3d74223e35c169, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/ab12c5e34e064f1c8504d5fe878add97, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/475193737fc24ed789fc466efe4b52bd, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/52f3560285f044ceb2059bf69522f0c9, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/f6c0276a42f14a699bd6cbe62d202a85, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/ed61dc61ec2e4471a8dfd8fadaf8c35e, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/56f748456cc046f1b054c6753401e26f, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/25a218c148c6407b9b153833c1f692c1] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp, totalSize=216.1 K 2023-04-20 12:56:20,493 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/5ddb5df9bd9c4ecdba8287d3ee2cbfca, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/d06e6a61e9444b7daf40d0040496eb54, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/015c558d946e464189962e779e9fa3a6, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/cbc4d02c68ed4708ae7e7e09ff150dc8, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/e9aa6376c3de4fd6b747f739fa0b5dfc, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/88803527119348b4ada46bd7f6e7cd18, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/3578e057e5e34551a249bf5d9f59fc6d, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/a318c92e343d48f9b09a4f791653b6c6] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp, totalSize=94.1 K 2023-04-20 12:56:20,494 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting ef945febbc434b91ba3d74223e35c169, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=119, earliestPutTs=1681995372539 2023-04-20 12:56:20,494 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 5ddb5df9bd9c4ecdba8287d3ee2cbfca, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=119, earliestPutTs=1681995372539 2023-04-20 12:56:20,494 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting ab12c5e34e064f1c8504d5fe878add97, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1681995372664 2023-04-20 12:56:20,494 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting d06e6a61e9444b7daf40d0040496eb54, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1681995372664 2023-04-20 12:56:20,494 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 015c558d946e464189962e779e9fa3a6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1681995373791 2023-04-20 12:56:20,494 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 475193737fc24ed789fc466efe4b52bd, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1681995373791 2023-04-20 12:56:20,495 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting cbc4d02c68ed4708ae7e7e09ff150dc8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=171, earliestPutTs=1681995374938 2023-04-20 12:56:20,495 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 52f3560285f044ceb2059bf69522f0c9, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=171, earliestPutTs=1681995374938 2023-04-20 12:56:20,495 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting e9aa6376c3de4fd6b747f739fa0b5dfc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1681995376062 2023-04-20 12:56:20,495 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 88803527119348b4ada46bd7f6e7cd18, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1681995377177 2023-04-20 12:56:20,496 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting f6c0276a42f14a699bd6cbe62d202a85, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1681995376062 2023-04-20 12:56:20,496 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 3578e057e5e34551a249bf5d9f59fc6d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1681995377501 2023-04-20 12:56:20,496 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting ed61dc61ec2e4471a8dfd8fadaf8c35e, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1681995377177 2023-04-20 12:56:20,496 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting a318c92e343d48f9b09a4f791653b6c6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1681995379640 2023-04-20 12:56:20,496 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 56f748456cc046f1b054c6753401e26f, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1681995377501 2023-04-20 12:56:20,497 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 25a218c148c6407b9b153833c1f692c1, keycount=250, bloomtype=ROW, size=38.9 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1681995379633 2023-04-20 12:56:20,509 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): 8113dc3e39e3420c86d60a8170c364f8#B#compaction#526 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:56:20,526 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8113dc3e39e3420c86d60a8170c364f8#A#compaction#527 average throughput is 1.23 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:56:20,529 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/B/a1f7f6b3890c470b98e2b304f9c4dd21 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/a1f7f6b3890c470b98e2b304f9c4dd21 2023-04-20 12:56:20,533 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 8113dc3e39e3420c86d60a8170c364f8/B of 8113dc3e39e3420c86d60a8170c364f8 into a1f7f6b3890c470b98e2b304f9c4dd21(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:56:20,533 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8113dc3e39e3420c86d60a8170c364f8: 2023-04-20 12:56:20,533 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8., storeName=8113dc3e39e3420c86d60a8170c364f8/B, priority=8, startTime=1681995380490; duration=0sec 2023-04-20 12:56:20,533 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:56:20,533 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-20 12:56:20,535 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 96378 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-20 12:56:20,535 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): 8113dc3e39e3420c86d60a8170c364f8/C is initiating minor compaction (all files) 2023-04-20 12:56:20,535 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8113dc3e39e3420c86d60a8170c364f8/C in TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:20,535 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/8041c0eb51e249009ed8d3e439bbc2b2, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/9990be75bc4241e0adbfbe1feedf6323, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/6bd9cd0da872410fb37968229f701cc8, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/2f4a9d359c4d429091ad9bf8373a4ff1, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/9913a77d6b1d424eb22bc682cb7290f7, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/c364e0615c134c7a8e30f977d2eee77a, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/13e1c545f4db47329cb58643a62e5f81, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/0a9c01fcbaf640f1b129e139d6cc7268] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp, totalSize=94.1 K 2023-04-20 12:56:20,536 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 8041c0eb51e249009ed8d3e439bbc2b2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=119, earliestPutTs=1681995372539 2023-04-20 12:56:20,536 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 9990be75bc4241e0adbfbe1feedf6323, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1681995372664 2023-04-20 12:56:20,536 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 6bd9cd0da872410fb37968229f701cc8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1681995373791 2023-04-20 12:56:20,536 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 2f4a9d359c4d429091ad9bf8373a4ff1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=171, earliestPutTs=1681995374938 2023-04-20 12:56:20,537 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 9913a77d6b1d424eb22bc682cb7290f7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1681995376062 2023-04-20 12:56:20,537 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting c364e0615c134c7a8e30f977d2eee77a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1681995377177 2023-04-20 12:56:20,537 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 13e1c545f4db47329cb58643a62e5f81, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1681995377501 2023-04-20 12:56:20,537 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 0a9c01fcbaf640f1b129e139d6cc7268, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1681995379640 2023-04-20 12:56:20,548 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): 8113dc3e39e3420c86d60a8170c364f8#C#compaction#528 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:56:20,570 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:20,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:20,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:20,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:20,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:20,573 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:20,573 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:20,573 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:20,573 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:20,573 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:20,574 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:20,574 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:20,574 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:20,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:20,574 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,574 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:20,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:20,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:20,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:20,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:20,575 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:20,575 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:20,575 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:20,575 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:20,575 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:20,575 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:20,575 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:20,575 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:20,575 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:20,576 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:20,576 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8113dc3e39e3420c86d60a8170c364f8 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-20 12:56:20,576 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=A 2023-04-20 12:56:20,576 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:20,576 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=B 2023-04-20 12:56:20,576 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:20,576 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=C 2023-04-20 12:56:20,576 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:20,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 8113dc3e39e3420c86d60a8170c364f8 2023-04-20 12:56:20,770 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:20,809 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:20,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41978 deadline: 1681995440803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,810 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:20,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42008 deadline: 1681995440803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,810 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:20,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42076 deadline: 1681995440804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,810 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:20,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42016 deadline: 1681995440804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,814 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:20,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42018 deadline: 1681995440810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,911 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:20,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41978 deadline: 1681995440910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,911 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:20,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42008 deadline: 1681995440911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,912 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:20,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42076 deadline: 1681995440911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,913 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:20,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42016 deadline: 1681995440911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,917 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:20,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42018 deadline: 1681995440915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:20,955 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/C/e1fcb8fccdd64dc1a626c87191d8f7b0 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/e1fcb8fccdd64dc1a626c87191d8f7b0 2023-04-20 12:56:20,959 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 8113dc3e39e3420c86d60a8170c364f8/C of 8113dc3e39e3420c86d60a8170c364f8 into e1fcb8fccdd64dc1a626c87191d8f7b0(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:56:20,959 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8113dc3e39e3420c86d60a8170c364f8: 2023-04-20 12:56:20,959 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8., storeName=8113dc3e39e3420c86d60a8170c364f8/C, priority=8, startTime=1681995380490; duration=0sec 2023-04-20 12:56:20,959 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:56:20,986 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420f013a9fc63d046489272b3332a838668 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420f013a9fc63d046489272b3332a838668 2023-04-20 12:56:20,990 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=252, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/A/8b128815343541178677c7e204c81111 2023-04-20 12:56:21,113 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:21,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42008 deadline: 1681995441112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:21,113 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:21,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41978 deadline: 1681995441113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:21,114 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:21,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42076 deadline: 1681995441113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:21,116 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:21,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42016 deadline: 1681995441115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:21,120 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:21,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42018 deadline: 1681995441118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:21,342 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/A/31e501f670fb43ce9795cef57068b463 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/31e501f670fb43ce9795cef57068b463 2023-04-20 12:56:21,345 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 8113dc3e39e3420c86d60a8170c364f8/A of 8113dc3e39e3420c86d60a8170c364f8 into 31e501f670fb43ce9795cef57068b463(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:56:21,345 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8113dc3e39e3420c86d60a8170c364f8: 2023-04-20 12:56:21,345 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8., storeName=8113dc3e39e3420c86d60a8170c364f8/A, priority=8, startTime=1681995380490; duration=0sec 2023-04-20 12:56:21,345 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:56:21,397 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=252 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/B/4df6e7895f434ffdad8ecad0b8a138b9 2023-04-20 12:56:21,416 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:21,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42076 deadline: 1681995441414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:21,418 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:21,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41978 deadline: 1681995441417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:21,418 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:21,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42008 deadline: 1681995441417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:21,419 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:21,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42016 deadline: 1681995441417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:21,421 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:21,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42018 deadline: 1681995441420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:21,813 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=252 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/C/7e7129915fa040cca85351312ac89940 2023-04-20 12:56:21,816 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/A/8b128815343541178677c7e204c81111 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/8b128815343541178677c7e204c81111 2023-04-20 12:56:21,819 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/8b128815343541178677c7e204c81111, entries=150, sequenceid=252, filesize=25.3 K 2023-04-20 12:56:21,819 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/B/4df6e7895f434ffdad8ecad0b8a138b9 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/4df6e7895f434ffdad8ecad0b8a138b9 2023-04-20 12:56:21,822 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/4df6e7895f434ffdad8ecad0b8a138b9, entries=150, sequenceid=252, filesize=11.8 K 2023-04-20 12:56:21,822 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/C/7e7129915fa040cca85351312ac89940 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/7e7129915fa040cca85351312ac89940 2023-04-20 12:56:21,825 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/7e7129915fa040cca85351312ac89940, entries=150, sequenceid=252, filesize=11.8 K 2023-04-20 12:56:21,825 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 8113dc3e39e3420c86d60a8170c364f8 in 1249ms, sequenceid=252, compaction requested=false 2023-04-20 12:56:21,825 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8113dc3e39e3420c86d60a8170c364f8: 2023-04-20 12:56:21,825 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:21,826 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:21,826 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:21,826 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:21,826 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:21,826 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:21,826 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:21,826 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:21,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:21,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:21,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:21,827 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:21,827 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:21,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:21,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:21,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:21,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:21,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:21,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:21,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@260c2b6f[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:21,828 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:21,828 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:21,828 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:21,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:21,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:21,828 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:21,828 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:21,829 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:21,829 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:21,829 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:21,829 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:21,829 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:21,829 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:21,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:21,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:21,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:21,829 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:21,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:21,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:21,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:21,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:21,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:21,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:21,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:21,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:21,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:21,831 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:21,831 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:21,831 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:21,831 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:21,831 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:21,832 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:21,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:21,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:21,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:21,832 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:21,832 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:21,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:21,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:21,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:21,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:21,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:21,832 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:21,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:21,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:21,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:21,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:21,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:21,833 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:21,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:21,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:21,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:21,835 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:21,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:21,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:21,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:21,835 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:21,835 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:21,835 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:21,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:21,835 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:21,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:21,835 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:21,835 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:21,835 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:21,835 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:21,835 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:21,835 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:21,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:21,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:21,835 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:21,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 8113dc3e39e3420c86d60a8170c364f8 2023-04-20 12:56:21,918 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8113dc3e39e3420c86d60a8170c364f8 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-20 12:56:21,918 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=A 2023-04-20 12:56:21,918 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:56:21,918 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=B 2023-04-20 12:56:21,918 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:56:21,918 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=C 2023-04-20 12:56:21,919 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:56:21,929 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:21,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41978 deadline: 1681995441926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:21,930 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:21,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42008 deadline: 1681995441927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:21,930 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:21,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42076 deadline: 1681995441927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:21,931 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:21,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42016 deadline: 1681995441928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:21,932 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:21,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42018 deadline: 1681995441929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:21,936 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:21,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:21,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:21,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:21,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:21,940 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:21,940 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:21,940 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:21,940 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:21,941 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:21,941 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:21,941 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:21,941 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:21,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:21,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:21,942 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:21,942 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:21,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:21,942 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:21,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:21,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:21,943 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:21,945 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:21,945 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:21,945 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:21,945 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:21,945 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:21,958 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:21,958 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:21,958 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:21,958 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:21,958 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:21,958 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:21,958 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:21,958 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:21,958 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:21,958 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:21,958 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:21,959 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:21,959 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:21,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:21,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:21,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:21,959 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:21,959 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:21,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:21,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:21,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:21,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:21,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:21,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:21,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@572b582c[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:21,961 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:21,961 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:21,961 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:21,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:21,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:21,961 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:21,961 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:21,961 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:21,961 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:21,961 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:21,962 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:21,962 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:21,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:21,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:21,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:21,962 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:21,962 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:21,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:21,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:21,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:21,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:21,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:21,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:21,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:21,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:21,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:21,964 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:21,964 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:21,964 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:21,964 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:21,964 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:21,965 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:21,965 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:21,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:21,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:21,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:21,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:21,965 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:21,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:21,966 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:21,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:21,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:21,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:21,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:21,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:21,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:21,966 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:21,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:21,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:21,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:21,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:21,968 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:21,968 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:21,968 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:21,968 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:21,968 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:21,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:21,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:21,968 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:21,968 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:21,968 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:21,968 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:21,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:21,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:21,968 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:21,968 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:21,968 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:21,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:21,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:21,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:22,032 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:22,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41978 deadline: 1681995442030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,032 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:22,032 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:22,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42076 deadline: 1681995442031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42008 deadline: 1681995442030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,036 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:22,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42016 deadline: 1681995442035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,040 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:22,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42018 deadline: 1681995442038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,068 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:22,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:22,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:22,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:22,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:22,071 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:22,071 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:22,071 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:22,071 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:22,072 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,072 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:22,072 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:22,072 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:22,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:22,073 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:22,073 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:22,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:22,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:22,073 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:22,073 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:22,073 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:22,073 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:22,073 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:22,073 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:22,073 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:22,074 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:22,074 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:22,074 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:22,074 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:22,074 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:22,074 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:22,074 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:22,074 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,074 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:22,074 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:22,075 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,075 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:22,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:22,075 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,075 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:22,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:22,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:22,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:22,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:22,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:22,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59bf8d30[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:22,076 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:22,076 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,077 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,077 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:22,077 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:22,077 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,077 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:22,077 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:22,077 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,077 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:22,078 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:22,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:22,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:22,078 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:22,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:22,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:22,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:22,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:22,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:22,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:22,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:22,080 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:22,080 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:22,080 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:22,080 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:22,080 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:22,081 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,081 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,081 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:22,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:22,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:22,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:22,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:22,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:22,082 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:22,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:22,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:22,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,083 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:22,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:22,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,084 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,084 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:22,084 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:22,084 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:22,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:22,084 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,084 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:22,084 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:22,084 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,084 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:22,084 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:22,084 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:22,084 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:22,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:22,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:22,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:22,185 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:22,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:22,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:22,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:22,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:22,187 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:22,187 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:22,187 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:22,187 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:22,188 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,188 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:22,189 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,189 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:22,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:22,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:22,189 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,190 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:22,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:22,190 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:22,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:22,190 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:22,191 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:22,191 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:22,191 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:22,191 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:22,191 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:22,191 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:22,191 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:22,191 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:22,191 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:22,191 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:22,191 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:22,191 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:22,191 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:22,191 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,191 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:22,191 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:22,192 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,192 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:22,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:22,192 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,192 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:22,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:22,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:22,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:22,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:22,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:22,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76bf803c[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:22,193 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:22,193 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,194 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,194 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:22,194 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:22,194 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,194 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:22,194 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:22,195 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,195 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:22,195 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:22,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:22,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:22,195 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:22,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:22,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:22,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:22,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:22,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:22,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:22,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:22,197 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:22,197 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:22,197 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:22,197 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:22,197 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:22,197 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,197 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:22,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:22,198 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:22,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:22,198 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:22,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:22,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:22,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:22,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:22,199 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:22,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:22,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,201 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,201 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:22,201 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:22,201 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:22,201 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:22,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:22,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:22,201 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:22,201 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:22,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:22,201 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,201 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,201 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,201 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:22,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:22,201 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:22,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:22,234 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:22,234 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:22,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42008 deadline: 1681995442233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41978 deadline: 1681995442233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,234 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:22,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42076 deadline: 1681995442234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,238 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:22,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42016 deadline: 1681995442237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,242 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:22,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42018 deadline: 1681995442242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,301 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:22,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:22,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:22,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:22,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:22,304 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:22,304 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:22,304 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:22,304 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:22,304 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,304 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:22,305 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,305 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:22,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:22,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:22,305 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,305 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:22,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:22,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:22,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:22,306 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:22,306 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:22,306 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:22,306 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:22,306 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:22,306 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:22,306 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:22,306 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:22,306 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:22,307 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:22,307 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:22,307 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:22,307 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:22,307 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,307 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:22,307 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:22,308 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,308 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:22,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:22,308 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,308 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:22,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:22,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:22,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:22,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:22,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:22,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31af9ad7[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:22,309 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:22,309 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,309 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,310 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:22,310 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:22,310 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,310 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:22,310 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:22,311 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,311 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:22,311 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:22,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:22,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:22,311 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:22,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:22,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:22,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:22,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:22,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:22,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:22,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:22,312 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:22,313 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:22,313 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:22,313 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:22,313 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:22,313 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,313 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:22,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:22,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,313 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:22,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:22,314 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:22,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:22,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:22,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:22,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:22,315 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:22,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:22,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,317 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,317 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:22,317 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:22,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:22,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:22,317 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:22,317 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,317 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,317 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,317 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:22,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:22,317 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:22,317 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:22,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:22,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:22,317 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:22,317 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:22,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:22,328 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304202c4399a7dc8547bda4bdd823f7ca2e5d to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304202c4399a7dc8547bda4bdd823f7ca2e5d 2023-04-20 12:56:22,417 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:22,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:22,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:22,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:22,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:22,420 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:22,420 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:22,420 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:22,420 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:22,420 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,421 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:22,421 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:22,421 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:22,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:22,421 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,421 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:22,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:22,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:22,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:22,422 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:22,422 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:22,422 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:22,422 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:22,422 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:22,422 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:22,422 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:22,422 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:22,422 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:22,423 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:22,423 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:22,423 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:22,423 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:22,423 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:22,423 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,423 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:22,423 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:22,424 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,424 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:22,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:22,424 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,424 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:22,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:22,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:22,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:22,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:22,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:22,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c485ef6[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:22,425 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:22,425 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,425 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,425 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:22,425 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:22,426 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:22,426 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,426 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:22,426 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,426 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:22,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:22,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:22,426 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:22,426 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:22,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:22,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:22,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:22,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:22,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:22,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:22,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:22,428 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:22,428 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:22,428 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:22,428 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:22,428 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:22,429 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,429 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,429 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:22,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:22,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:22,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:22,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:22,429 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:22,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:22,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:22,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:22,430 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:22,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:22,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,432 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,432 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:22,432 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:22,432 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:22,432 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:22,432 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,432 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,432 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:22,432 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:22,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:22,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:22,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:22,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:22,432 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,432 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:22,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:22,432 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:22,432 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:22,532 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:22,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:22,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:22,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:22,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:22,535 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:22,535 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:22,535 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:22,535 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:22,535 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,536 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:22,536 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:22,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:22,536 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:22,537 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,537 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:22,537 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:22,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41978 deadline: 1681995442536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:22,537 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:22,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:22,537 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:22,538 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:22,538 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:22,538 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:22,538 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:22,538 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:22,538 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:22,538 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:22,538 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:22,538 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:22,538 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:22,538 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:22,538 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:22,538 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:22,538 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,538 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:22,538 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:22,538 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:22,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42008 deadline: 1681995442537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,538 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:22,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42076 deadline: 1681995442537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,539 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:22,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42016 deadline: 1681995442539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,539 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:22,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:22,539 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:22,540 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,540 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:22,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:22,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:22,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:22,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:22,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19bb9242[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:22,541 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:22,541 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,542 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,542 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:22,542 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:22,542 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:22,542 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,542 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:22,543 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,543 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:22,543 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:22,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:22,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:22,543 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:22,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:22,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:22,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:22,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:22,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:22,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:22,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:22,545 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:22,545 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:22,545 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:22,545 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:22,545 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:22,546 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,546 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:22,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:22,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,546 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:22,546 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:22,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:22,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:22,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:22,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,547 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:22,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:22,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42018 deadline: 1681995442545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:22,547 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:22,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:22,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,548 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,548 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:22,548 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:22,549 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:22,549 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:22,549 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:22,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:22,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:22,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:22,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:22,549 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:22,549 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,549 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:22,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:22,549 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:22,549 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:22,549 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,549 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,649 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:22,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:22,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:22,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:22,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:22,651 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:22,651 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:22,651 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:22,651 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:22,652 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,652 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:22,653 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,653 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:22,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:22,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:22,653 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,653 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:22,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:22,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:22,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:22,653 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:22,654 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:22,654 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:22,654 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:22,654 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:22,654 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:22,654 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:22,654 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:22,654 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:22,654 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:22,654 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:22,654 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:22,654 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:22,654 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:22,654 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,654 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:22,654 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:22,655 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,655 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:22,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:22,655 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,655 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:22,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:22,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:22,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:22,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:22,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:22,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@130f5f29[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:22,656 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:22,656 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,657 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,657 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:22,657 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:22,657 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:22,657 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,657 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:22,657 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:22,657 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,657 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:22,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:22,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:22,658 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:22,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:22,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:22,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:22,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:22,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:22,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:22,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:22,659 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:22,659 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:22,659 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:22,659 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:22,659 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:22,660 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:22,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:22,660 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,660 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:22,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,660 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:22,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:22,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:22,661 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:22,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:22,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:22,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,662 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:22,662 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:22,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:22,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:22,662 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,663 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:22,663 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:22,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:22,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:22,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:22,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:22,663 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:22,663 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:22,663 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,663 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,663 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,663 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:22,663 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:22,663 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:22,735 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=275, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/A/cdc3c7dcae39452e960a8cddfeaffb7a 2023-04-20 12:56:22,763 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:22,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:22,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:22,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:22,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:22,766 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:22,766 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:22,766 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:22,767 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:22,767 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,767 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:22,767 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:22,768 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:22,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:22,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:22,768 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,768 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:22,768 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:22,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:22,768 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:22,768 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:22,768 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:22,769 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:22,769 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:22,769 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:22,769 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:22,769 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:22,769 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:22,769 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:22,769 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:22,769 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:22,769 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:22,769 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:22,769 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,769 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:22,769 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:22,770 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,770 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:22,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:22,770 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,770 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:22,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:22,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:22,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:22,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:22,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:22,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ebb16bd[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:22,771 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:22,771 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,772 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,772 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:22,772 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:22,772 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:22,772 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,772 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:22,772 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,772 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:22,772 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:22,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:22,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:22,773 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:22,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:22,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:22,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:22,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:22,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:22,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:22,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:22,775 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:22,775 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:22,775 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:22,775 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:22,775 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:22,775 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:22,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:22,775 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,776 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:22,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:22,776 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:22,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:22,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:22,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:22,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:22,777 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:22,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:22,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,778 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,778 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:22,778 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:22,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:22,778 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:22,778 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:22,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:22,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:22,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:22,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:22,778 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:22,779 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,779 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,779 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,779 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:22,779 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:22,779 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:22,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:22,879 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:22,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:22,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:22,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:22,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:22,881 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:22,881 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:22,881 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:22,881 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:22,881 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,882 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:22,882 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:22,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:22,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:22,882 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,882 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,882 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:22,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:22,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:22,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:22,883 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:22,883 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:22,883 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:22,883 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:22,883 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:22,883 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:22,883 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:22,883 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:22,883 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:22,884 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:22,884 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:22,884 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:22,884 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:22,884 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:22,884 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,884 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:22,884 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:22,885 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,885 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:22,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:22,885 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,885 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:22,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:22,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:22,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:22,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:22,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:22,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20fa4c72[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:22,886 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:22,886 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,887 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,887 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:22,887 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,887 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:22,887 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:22,887 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:22,888 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,888 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:22,888 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:22,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:22,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:22,888 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:22,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:22,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:22,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:22,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:22,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:22,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:22,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:22,890 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:22,890 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:22,890 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:22,890 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:22,890 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:22,891 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,891 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:22,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:22,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,891 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:22,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:22,891 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:22,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:22,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:22,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:22,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:22,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,892 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:22,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:22,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,894 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,894 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:22,894 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:22,894 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:22,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:22,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:22,894 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:22,894 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:22,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:22,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:22,894 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:22,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:22,894 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,894 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:22,895 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:22,894 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:22,895 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,995 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:22,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:22,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:22,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:22,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:22,997 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:22,997 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:22,997 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:22,997 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:22,997 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,997 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:22,998 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,998 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:22,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:22,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:22,998 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:22,998 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:22,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:22,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:22,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:22,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:22,999 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:22,999 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:22,999 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:22,999 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:22,999 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:22,999 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:22,999 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:22,999 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:22,999 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:22,999 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:22,999 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:22,999 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:22,999 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:22,999 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:22,999 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,000 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:23,000 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:23,000 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,000 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:23,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:23,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:23,000 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:23,000 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:23,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:23,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:23,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:23,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:23,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:23,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6596f3ad[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:23,002 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:23,002 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:23,002 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:23,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:23,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:23,002 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:23,003 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:23,003 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:23,003 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,003 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:23,003 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,003 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:23,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:23,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:23,003 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:23,003 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:23,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:23,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:23,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:23,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:23,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:23,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:23,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:23,005 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:23,005 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:23,005 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:23,005 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:23,005 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:23,006 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:23,006 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:23,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:23,006 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:23,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:23,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:23,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:23,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:23,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:23,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:23,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:23,006 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:23,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:23,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:23,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:23,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,007 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:23,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:23,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,008 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,008 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:23,008 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:23,008 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:23,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:23,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:23,008 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:23,008 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:23,008 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:23,008 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:23,008 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:23,009 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:23,008 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:23,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:23,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:23,009 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:23,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:23,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:23,040 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:23,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41978 deadline: 1681995443039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,043 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:23,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42076 deadline: 1681995443041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,044 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:23,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42016 deadline: 1681995443042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,044 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:23,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42008 deadline: 1681995443042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,049 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:23,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42018 deadline: 1681995443048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,109 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:23,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:23,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:23,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:23,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:23,111 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:23,111 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:23,111 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:23,111 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:23,111 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:23,111 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:23,112 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,112 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:23,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:23,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:23,112 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,112 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:23,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:23,113 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:23,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:23,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:23,113 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:23,113 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:23,113 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:23,113 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:23,113 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:23,113 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:23,113 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:23,113 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:23,113 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:23,113 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:23,114 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:23,114 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:23,114 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:23,114 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:23,114 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,114 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:23,114 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:23,114 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,114 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:23,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:23,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:23,115 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:23,115 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:23,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:23,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:23,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:23,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:23,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:23,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c890e21[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:23,116 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:23,116 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:23,117 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:23,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:23,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:23,117 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:23,117 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,117 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:23,117 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:23,117 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:23,118 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,118 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:23,118 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:23,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:23,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:23,118 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:23,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:23,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:23,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:23,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:23,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:23,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:23,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:23,119 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:23,119 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:23,119 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:23,119 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:23,119 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:23,120 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:23,120 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:23,120 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:23,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:23,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:23,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:23,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:23,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:23,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:23,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:23,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:23,121 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:23,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:23,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:23,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:23,122 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:23,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:23,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,123 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:23,123 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:23,123 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:23,123 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:23,123 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:23,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:23,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:23,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:23,123 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:23,123 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:23,123 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:23,123 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:23,123 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:23,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:23,123 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:23,124 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:23,124 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:23,142 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=275 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/B/c987e8c0f81c44dd84258d02a5034b2c 2023-04-20 12:56:23,224 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:23,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:23,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:23,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:23,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:23,226 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:23,226 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:23,226 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:23,226 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:23,227 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:23,227 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:23,227 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,227 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:23,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:23,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:23,228 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,228 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:23,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:23,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:23,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:23,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:23,228 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:23,228 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:23,228 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:23,228 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:23,228 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:23,228 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:23,229 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:23,229 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:23,229 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:23,229 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:23,229 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:23,229 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:23,229 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:23,229 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:23,229 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,229 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:23,229 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:23,230 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,230 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:23,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:23,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:23,230 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:23,230 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:23,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:23,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:23,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:23,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:23,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:23,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a01fff3[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:23,231 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:23,231 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:23,232 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:23,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:23,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:23,232 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:23,232 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:23,232 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,232 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:23,232 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:23,232 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:23,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:23,232 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:23,233 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:23,233 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:23,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:23,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:23,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:23,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:23,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:23,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:23,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:23,234 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:23,234 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:23,234 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:23,234 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:23,234 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:23,235 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:23,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:23,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:23,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:23,235 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:23,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:23,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:23,235 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:23,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:23,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:23,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:23,235 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:23,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:23,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:23,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:23,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,236 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:23,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:23,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,239 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:23,239 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:23,239 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:23,239 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:23,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:23,239 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:23,239 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:23,239 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,239 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:23,239 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:23,239 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:23,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:23,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:23,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:23,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:23,240 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:23,240 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:23,340 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:23,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:23,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:23,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:23,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:23,343 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:23,343 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:23,343 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:23,343 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:23,343 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:23,343 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:23,344 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:23,344 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:23,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:23,344 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,344 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:23,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:23,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:23,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:23,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:23,344 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:23,345 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:23,345 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:23,345 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:23,345 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:23,345 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:23,345 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:23,345 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:23,345 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:23,345 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:23,345 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:23,345 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:23,345 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:23,345 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:23,345 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,345 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:23,345 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:23,346 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,346 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:23,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:23,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:23,346 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:23,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:23,346 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:23,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:23,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:23,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:23,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:23,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78eaab1d[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:23,347 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:23,348 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:23,348 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:23,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:23,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:23,348 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:23,348 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:23,348 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:23,348 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,348 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:23,349 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:23,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:23,349 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:23,349 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:23,349 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:23,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:23,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:23,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:23,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:23,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:23,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:23,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:23,351 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:23,351 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:23,351 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:23,351 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:23,351 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:23,352 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:23,352 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:23,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:23,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:23,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:23,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:23,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:23,352 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:23,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:23,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:23,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:23,353 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:23,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:23,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:23,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:23,353 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:23,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:23,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,355 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,355 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:23,355 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:23,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:23,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:23,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:23,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:23,355 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:23,355 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:23,355 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:23,355 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:23,355 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:23,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:23,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:23,355 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:23,355 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,355 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:23,355 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:23,355 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:23,455 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:23,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:23,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:23,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:23,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:23,458 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:23,458 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:23,458 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:23,458 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:23,458 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:23,458 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:23,459 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:23,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:23,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:23,459 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:23,459 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,459 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:23,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:23,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:23,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:23,460 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:23,460 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:23,460 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:23,460 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:23,460 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:23,460 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:23,460 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:23,460 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:23,460 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:23,460 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:23,460 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:23,460 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:23,460 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:23,460 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:23,460 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,460 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:23,460 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:23,461 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,461 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:23,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:23,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:23,461 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:23,462 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:23,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:23,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:23,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:23,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:23,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:23,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ab32436[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:23,463 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:23,463 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:23,463 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:23,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:23,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:23,464 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:23,464 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:23,464 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:23,465 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,465 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:23,465 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,465 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:23,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:23,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:23,465 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:23,466 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:23,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:23,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:23,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:23,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:23,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:23,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:23,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:23,467 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:23,467 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:23,467 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:23,467 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:23,467 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:23,468 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:23,468 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:23,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:23,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:23,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:23,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:23,468 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:23,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:23,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:23,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:23,469 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:23,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:23,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:23,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:23,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:23,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,470 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:23,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:23,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,471 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,471 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:23,471 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:23,471 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:23,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:23,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:23,471 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,471 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:23,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:23,471 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:23,471 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:23,471 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:23,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:23,471 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:23,471 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:23,471 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:23,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:23,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:23,550 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=275 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/C/4b53723f1d794214b29548db1e5ee2fa 2023-04-20 12:56:23,553 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/A/cdc3c7dcae39452e960a8cddfeaffb7a as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/cdc3c7dcae39452e960a8cddfeaffb7a 2023-04-20 12:56:23,556 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/cdc3c7dcae39452e960a8cddfeaffb7a, entries=200, sequenceid=275, filesize=32.3 K 2023-04-20 12:56:23,556 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/B/c987e8c0f81c44dd84258d02a5034b2c as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/c987e8c0f81c44dd84258d02a5034b2c 2023-04-20 12:56:23,559 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/c987e8c0f81c44dd84258d02a5034b2c, entries=150, sequenceid=275, filesize=11.9 K 2023-04-20 12:56:23,559 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/C/4b53723f1d794214b29548db1e5ee2fa as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/4b53723f1d794214b29548db1e5ee2fa 2023-04-20 12:56:23,561 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/4b53723f1d794214b29548db1e5ee2fa, entries=150, sequenceid=275, filesize=11.9 K 2023-04-20 12:56:23,562 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 8113dc3e39e3420c86d60a8170c364f8 in 1644ms, sequenceid=275, compaction requested=true 2023-04-20 12:56:23,562 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8113dc3e39e3420c86d60a8170c364f8: 2023-04-20 12:56:23,562 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:56:23,562 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:56:23,562 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:56:23,562 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:56:23,562 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:56:23,564 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85390 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:56:23,564 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): 8113dc3e39e3420c86d60a8170c364f8/A is initiating minor compaction (all files) 2023-04-20 12:56:23,564 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36770 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:56:23,564 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8113dc3e39e3420c86d60a8170c364f8/A in TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:23,564 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): 8113dc3e39e3420c86d60a8170c364f8/B is initiating minor compaction (all files) 2023-04-20 12:56:23,564 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/31e501f670fb43ce9795cef57068b463, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/8b128815343541178677c7e204c81111, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/cdc3c7dcae39452e960a8cddfeaffb7a] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp, totalSize=83.4 K 2023-04-20 12:56:23,564 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8113dc3e39e3420c86d60a8170c364f8/B in TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:23,564 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/a1f7f6b3890c470b98e2b304f9c4dd21, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/4df6e7895f434ffdad8ecad0b8a138b9, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/c987e8c0f81c44dd84258d02a5034b2c] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp, totalSize=35.9 K 2023-04-20 12:56:23,564 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 31e501f670fb43ce9795cef57068b463, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1681995379640 2023-04-20 12:56:23,564 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting a1f7f6b3890c470b98e2b304f9c4dd21, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1681995379640 2023-04-20 12:56:23,565 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 8b128815343541178677c7e204c81111, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=252, earliestPutTs=1681995379654 2023-04-20 12:56:23,565 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 4df6e7895f434ffdad8ecad0b8a138b9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=252, earliestPutTs=1681995379654 2023-04-20 12:56:23,565 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting cdc3c7dcae39452e960a8cddfeaffb7a, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1681995380783 2023-04-20 12:56:23,565 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting c987e8c0f81c44dd84258d02a5034b2c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1681995380783 2023-04-20 12:56:23,570 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): 8113dc3e39e3420c86d60a8170c364f8#B#compaction#535 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:56:23,571 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:23,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:23,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:23,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:23,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:23,576 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:23,576 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:23,576 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:23,576 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:23,577 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:23,577 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:23,577 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,577 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:23,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:23,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:23,577 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8113dc3e39e3420c86d60a8170c364f8#A#compaction#536 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:56:23,577 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:23,578 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:23,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:23,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:23,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:23,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:23,579 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:23,579 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:23,579 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:23,579 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:23,579 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:23,579 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:23,580 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:23,580 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:23,580 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:23,580 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:23,580 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8113dc3e39e3420c86d60a8170c364f8 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-20 12:56:23,580 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=A 2023-04-20 12:56:23,580 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:23,581 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=B 2023-04-20 12:56:23,581 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:23,581 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=C 2023-04-20 12:56:23,581 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:23,597 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420ff71795432f74a24b9e6a6af0a1dc633 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420ff71795432f74a24b9e6a6af0a1dc633 2023-04-20 12:56:23,986 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/B/d7e4c80585cc443a967ec8134d54d9ec as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/d7e4c80585cc443a967ec8134d54d9ec 2023-04-20 12:56:23,996 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8113dc3e39e3420c86d60a8170c364f8/B of 8113dc3e39e3420c86d60a8170c364f8 into d7e4c80585cc443a967ec8134d54d9ec(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:56:23,996 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8113dc3e39e3420c86d60a8170c364f8: 2023-04-20 12:56:23,996 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8., storeName=8113dc3e39e3420c86d60a8170c364f8/B, priority=13, startTime=1681995383562; duration=0sec 2023-04-20 12:56:23,996 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:56:23,996 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-20 12:56:23,997 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36770 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-20 12:56:23,997 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): 8113dc3e39e3420c86d60a8170c364f8/C is initiating minor compaction (all files) 2023-04-20 12:56:23,997 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8113dc3e39e3420c86d60a8170c364f8/C in TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:23,997 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/e1fcb8fccdd64dc1a626c87191d8f7b0, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/7e7129915fa040cca85351312ac89940, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/4b53723f1d794214b29548db1e5ee2fa] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp, totalSize=35.9 K 2023-04-20 12:56:23,997 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting e1fcb8fccdd64dc1a626c87191d8f7b0, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1681995379640 2023-04-20 12:56:23,997 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 7e7129915fa040cca85351312ac89940, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=252, earliestPutTs=1681995379654 2023-04-20 12:56:23,998 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 4b53723f1d794214b29548db1e5ee2fa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1681995380783 2023-04-20 12:56:24,001 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=286, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/A/7655abce4c414d548981b78ffc8237e3 2023-04-20 12:56:24,002 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): 8113dc3e39e3420c86d60a8170c364f8#C#compaction#538 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:56:24,008 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=286 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/B/63d18988c9564d73a631864144a0660f 2023-04-20 12:56:24,010 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/C/4a7bb375712d44ef9e765a27951db3cd as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/4a7bb375712d44ef9e765a27951db3cd 2023-04-20 12:56:24,013 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8113dc3e39e3420c86d60a8170c364f8/C of 8113dc3e39e3420c86d60a8170c364f8 into 4a7bb375712d44ef9e765a27951db3cd(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:56:24,013 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8113dc3e39e3420c86d60a8170c364f8: 2023-04-20 12:56:24,013 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8., storeName=8113dc3e39e3420c86d60a8170c364f8/C, priority=13, startTime=1681995383562; duration=0sec 2023-04-20 12:56:24,014 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:56:24,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 8113dc3e39e3420c86d60a8170c364f8 2023-04-20 12:56:24,053 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:24,069 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:24,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42016 deadline: 1681995444066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:24,070 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:24,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42008 deadline: 1681995444066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:24,070 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:24,070 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681995384065, max timestamp=1681995384069Num uniques 859; 2023-04-20 12:56:24,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41978 deadline: 1681995444067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:24,070 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681995384065, max timestamp=1681995384069Num uniques 859; 2023-04-20 12:56:24,070 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 12:56:24,070 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681995384065, max timestamp=1681995384069Num uniques 859; 2023-04-20 12:56:24,070 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 12:56:24,070 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-20 12:56:24,092 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:24,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42018 deadline: 1681995444089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:24,093 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:24,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42076 deadline: 1681995444091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:24,172 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:24,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42016 deadline: 1681995444170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:24,172 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:24,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42008 deadline: 1681995444171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:24,172 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:24,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41978 deadline: 1681995444171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:24,194 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:24,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42018 deadline: 1681995444193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:24,194 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:24,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42076 deadline: 1681995444194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:24,375 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:24,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42016 deadline: 1681995444373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:24,375 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:24,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41978 deadline: 1681995444373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:24,375 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:24,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42008 deadline: 1681995444374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:24,394 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/A/b27365ec9cea4e778caa5468f04c0e04 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/b27365ec9cea4e778caa5468f04c0e04 2023-04-20 12:56:24,396 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8113dc3e39e3420c86d60a8170c364f8/A of 8113dc3e39e3420c86d60a8170c364f8 into b27365ec9cea4e778caa5468f04c0e04(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:56:24,397 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8113dc3e39e3420c86d60a8170c364f8: 2023-04-20 12:56:24,397 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8., storeName=8113dc3e39e3420c86d60a8170c364f8/A, priority=13, startTime=1681995383562; duration=0sec 2023-04-20 12:56:24,397 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:56:24,398 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:24,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42076 deadline: 1681995444396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:24,398 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:24,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42018 deadline: 1681995444396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:24,416 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=286 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/C/6ce235d43bba43668b6429741b429a69 2023-04-20 12:56:24,419 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/A/7655abce4c414d548981b78ffc8237e3 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/7655abce4c414d548981b78ffc8237e3 2023-04-20 12:56:24,422 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/7655abce4c414d548981b78ffc8237e3, entries=100, sequenceid=286, filesize=18.6 K 2023-04-20 12:56:24,422 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/B/63d18988c9564d73a631864144a0660f as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/63d18988c9564d73a631864144a0660f 2023-04-20 12:56:24,425 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/63d18988c9564d73a631864144a0660f, entries=100, sequenceid=286, filesize=9.5 K 2023-04-20 12:56:24,426 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/C/6ce235d43bba43668b6429741b429a69 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/6ce235d43bba43668b6429741b429a69 2023-04-20 12:56:24,429 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/6ce235d43bba43668b6429741b429a69, entries=100, sequenceid=286, filesize=9.5 K 2023-04-20 12:56:24,429 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 8113dc3e39e3420c86d60a8170c364f8 in 849ms, sequenceid=286, compaction requested=false 2023-04-20 12:56:24,430 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8113dc3e39e3420c86d60a8170c364f8: 2023-04-20 12:56:24,430 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:24,430 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:24,430 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:24,430 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:24,430 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:24,430 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:24,431 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:24,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:24,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:24,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:24,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:24,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:24,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:24,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:24,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:24,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:24,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e0fcf47[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:24,432 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:24,432 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:24,433 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:24,433 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:24,438 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:24,439 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:24,439 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:24,439 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:24,439 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:24,439 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:24,439 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:24,449 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:24,449 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:24,449 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:24,449 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:24,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:24,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:24,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:24,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:24,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:24,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:24,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:24,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:24,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:24,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:24,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:24,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:24,451 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:24,451 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:24,451 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:24,451 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:24,451 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:24,454 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:24,454 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:24,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:24,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:24,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:24,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:24,455 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:24,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:24,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:24,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:24,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:24,455 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:24,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:24,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:24,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:24,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:24,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:24,456 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:24,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:24,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:24,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:24,458 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:24,458 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:24,458 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:24,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:24,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:24,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:24,458 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:24,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:24,458 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:24,458 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:24,459 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:24,459 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:24,458 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:24,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:24,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:24,459 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:24,459 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:24,459 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:24,459 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:24,559 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:24,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:24,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:24,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:24,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:24,561 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:24,561 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:24,561 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:24,561 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:24,561 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:24,561 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:24,562 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:24,562 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:24,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:24,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:24,562 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:24,562 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:24,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:24,563 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:24,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:24,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:24,563 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:24,563 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:24,563 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:24,563 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:24,563 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:24,563 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:24,563 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:24,563 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:24,563 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:24,563 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:24,564 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8113dc3e39e3420c86d60a8170c364f8 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-20 12:56:24,564 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=A 2023-04-20 12:56:24,564 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:56:24,564 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=B 2023-04-20 12:56:24,564 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:56:24,564 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=C 2023-04-20 12:56:24,564 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:56:24,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 8113dc3e39e3420c86d60a8170c364f8 2023-04-20 12:56:24,679 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:24,686 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:24,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42008 deadline: 1681995444684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:24,688 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:24,688 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:24,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41978 deadline: 1681995444685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:24,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42016 deadline: 1681995444686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:24,701 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:24,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42076 deadline: 1681995444699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:24,701 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:24,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42018 deadline: 1681995444699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:24,788 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:24,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42008 deadline: 1681995444787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:24,790 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:24,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41978 deadline: 1681995444789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:24,791 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:24,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42016 deadline: 1681995444789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:24,973 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420b7ea28ffbcb1438fb92aa2cd67b77bbb to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420b7ea28ffbcb1438fb92aa2cd67b77bbb 2023-04-20 12:56:24,977 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=324, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/A/babf334946df4b5ebc1dc568354dfb26 2023-04-20 12:56:24,991 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:24,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42008 deadline: 1681995444990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:24,993 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=324 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/B/fca31bc4c2094bc680460ce08374c333 2023-04-20 12:56:24,994 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:24,994 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:24,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42016 deadline: 1681995444991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:24,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41978 deadline: 1681995444992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:25,205 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:25,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42076 deadline: 1681995445205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:25,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:25,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42018 deadline: 1681995445206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:25,292 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:25,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42008 deadline: 1681995445292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:25,297 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:25,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42016 deadline: 1681995445296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:25,298 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:25,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41978 deadline: 1681995445297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:25,400 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=324 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/C/7a43003ac84444dab85617dd65e1dc56 2023-04-20 12:56:25,403 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/A/babf334946df4b5ebc1dc568354dfb26 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/babf334946df4b5ebc1dc568354dfb26 2023-04-20 12:56:25,406 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/babf334946df4b5ebc1dc568354dfb26, entries=150, sequenceid=324, filesize=25.4 K 2023-04-20 12:56:25,407 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/B/fca31bc4c2094bc680460ce08374c333 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/fca31bc4c2094bc680460ce08374c333 2023-04-20 12:56:25,409 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/fca31bc4c2094bc680460ce08374c333, entries=150, sequenceid=324, filesize=11.9 K 2023-04-20 12:56:25,410 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/C/7a43003ac84444dab85617dd65e1dc56 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/7a43003ac84444dab85617dd65e1dc56 2023-04-20 12:56:25,412 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/7a43003ac84444dab85617dd65e1dc56, entries=150, sequenceid=324, filesize=11.9 K 2023-04-20 12:56:25,413 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=60.38 KB/61830 for 8113dc3e39e3420c86d60a8170c364f8 in 849ms, sequenceid=324, compaction requested=true 2023-04-20 12:56:25,413 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8113dc3e39e3420c86d60a8170c364f8: 2023-04-20 12:56:25,413 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:25,413 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:25,413 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:25,413 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:25,413 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:25,413 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:25,414 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:25,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:25,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:25,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:25,414 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:25,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:25,414 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:25,414 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:25,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:25,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:25,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:25,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:25,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:25,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b744b12[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:25,416 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:25,416 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:25,416 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:25,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:25,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:25,416 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:25,416 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:25,416 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:25,417 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:25,417 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:25,417 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:25,417 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:25,417 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:25,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:25,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:25,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:25,417 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:25,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:25,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:25,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:25,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:25,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:25,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:25,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:25,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:25,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:25,419 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:25,419 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:25,419 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:25,419 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:25,419 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:25,420 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:25,420 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:25,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:25,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:25,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:25,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:25,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:25,420 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:25,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:25,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:25,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:25,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:25,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:25,421 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:25,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:25,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:25,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:25,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:25,422 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:25,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:25,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:25,423 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:25,424 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:25,424 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:25,424 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:25,424 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:25,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:25,424 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:25,424 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:25,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:25,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:25,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:25,424 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:25,424 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:25,424 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:25,424 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:25,424 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:25,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:25,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:25,524 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:25,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:25,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:25,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:25,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:25,527 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:25,527 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:25,527 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:25,527 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:25,527 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:25,527 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:25,528 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:25,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:25,528 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:25,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:25,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:25,528 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:25,528 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:25,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:25,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:25,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:25,529 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:25,529 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:25,529 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:25,529 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:25,529 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:25,529 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:25,529 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:25,529 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:25,529 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:25,529 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:25,529 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8113dc3e39e3420c86d60a8170c364f8 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-20 12:56:25,529 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=A 2023-04-20 12:56:25,530 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:25,530 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=B 2023-04-20 12:56:25,530 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:25,530 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=C 2023-04-20 12:56:25,530 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:25,539 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304200bfa43cbf72b49bd8c7ce87cc4f90208 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304200bfa43cbf72b49bd8c7ce87cc4f90208 2023-04-20 12:56:25,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 8113dc3e39e3420c86d60a8170c364f8 2023-04-20 12:56:25,798 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:25,815 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:25,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42016 deadline: 1681995445812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:25,815 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:25,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42008 deadline: 1681995445813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:25,817 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:25,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41978 deadline: 1681995445815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:25,917 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:25,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42016 deadline: 1681995445916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:25,917 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:25,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42008 deadline: 1681995445916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:25,919 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:25,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41978 deadline: 1681995445918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:25,943 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=336, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/A/031df4b6096c4b6483d13ceb6d43ef6b 2023-04-20 12:56:25,950 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=336 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/B/0ff9d1fb4c4f4456882e0f4285f4ff8d 2023-04-20 12:56:26,120 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:26,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42016 deadline: 1681995446118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,120 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:26,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42008 deadline: 1681995446119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,122 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:26,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41978 deadline: 1681995446120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,210 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:26,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42018 deadline: 1681995446207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,211 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:26,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42076 deadline: 1681995446209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,358 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=336 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/C/023026c42ddd434aa29bfb761265b1e8 2023-04-20 12:56:26,361 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/A/031df4b6096c4b6483d13ceb6d43ef6b as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/031df4b6096c4b6483d13ceb6d43ef6b 2023-04-20 12:56:26,363 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/031df4b6096c4b6483d13ceb6d43ef6b, entries=150, sequenceid=336, filesize=25.4 K 2023-04-20 12:56:26,364 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/B/0ff9d1fb4c4f4456882e0f4285f4ff8d as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/0ff9d1fb4c4f4456882e0f4285f4ff8d 2023-04-20 12:56:26,367 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/0ff9d1fb4c4f4456882e0f4285f4ff8d, entries=150, sequenceid=336, filesize=11.9 K 2023-04-20 12:56:26,367 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/C/023026c42ddd434aa29bfb761265b1e8 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/023026c42ddd434aa29bfb761265b1e8 2023-04-20 12:56:26,370 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/023026c42ddd434aa29bfb761265b1e8, entries=150, sequenceid=336, filesize=11.9 K 2023-04-20 12:56:26,370 INFO [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 8113dc3e39e3420c86d60a8170c364f8 in 841ms, sequenceid=336, compaction requested=true 2023-04-20 12:56:26,370 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8113dc3e39e3420c86d60a8170c364f8: 2023-04-20 12:56:26,370 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:26,370 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:26,370 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:26,370 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,370 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:26,370 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:26,371 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,371 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:26,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:26,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:26,372 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:26,372 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:26,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:26,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:26,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:26,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:26,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:26,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@466bb722[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:26,373 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:26,373 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:26,373 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:26,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:26,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:26,373 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:26,373 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:26,373 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,373 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:26,373 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:26,374 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:26,374 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:26,374 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:26,374 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:26,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:26,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:26,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:26,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:26,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:26,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:26,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:26,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:26,376 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:26,376 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:26,376 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:26,376 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:26,376 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:26,377 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:26,377 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:26,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:26,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:26,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:26,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:26,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:26,377 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:26,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:26,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:26,377 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:26,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:26,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:26,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:26,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:26,378 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:26,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:26,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,380 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,380 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:26,380 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:26,380 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:26,380 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:26,380 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,380 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:26,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:26,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:26,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:26,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:26,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:26,380 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:26,380 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:26,380 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:26,380 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:26,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:26,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:26,380 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:26,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 8113dc3e39e3420c86d60a8170c364f8 2023-04-20 12:56:26,424 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8113dc3e39e3420c86d60a8170c364f8 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-20 12:56:26,425 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=A 2023-04-20 12:56:26,425 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:56:26,425 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=B 2023-04-20 12:56:26,425 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:56:26,425 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=C 2023-04-20 12:56:26,425 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-20 12:56:26,434 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:26,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41978 deadline: 1681995446432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,435 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:26,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42016 deadline: 1681995446432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,435 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:26,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42008 deadline: 1681995446433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,480 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:26,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:26,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:26,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:26,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:26,483 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:26,483 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:26,483 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:26,483 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:26,483 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:26,483 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:26,484 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:26,484 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:26,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:26,484 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:26,484 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:26,484 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:26,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:26,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:26,485 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:26,485 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:26,485 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:26,485 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:26,485 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:26,485 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:26,485 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:26,485 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:26,485 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:26,485 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:26,485 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:26,485 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:26,485 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:26,485 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:26,485 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,485 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:26,485 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:26,486 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,486 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:26,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:26,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:26,486 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:26,486 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:26,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:26,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:26,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:26,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:26,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:26,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75caa94d[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:26,487 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:26,487 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:26,488 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:26,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:26,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:26,488 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:26,488 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:26,488 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,488 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:26,488 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:26,489 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,489 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:26,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:26,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:26,489 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:26,489 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:26,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:26,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:26,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:26,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:26,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:26,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:26,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:26,490 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:26,490 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:26,490 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:26,490 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:26,490 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:26,491 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:26,491 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:26,491 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:26,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:26,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:26,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:26,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:26,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:26,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:26,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:26,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:26,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:26,492 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:26,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:26,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:26,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,493 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:26,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:26,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,494 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,494 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:26,494 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:26,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:26,494 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:26,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:26,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:26,494 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:26,494 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:26,494 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:26,494 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:26,494 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:26,494 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:26,494 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:26,494 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:26,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:26,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:26,537 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:26,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41978 deadline: 1681995446535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,537 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:26,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42016 deadline: 1681995446535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,537 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:26,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42008 deadline: 1681995446535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,594 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:26,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:26,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:26,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:26,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:26,597 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:26,597 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:26,597 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:26,597 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:26,598 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:26,598 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:26,599 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,599 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:26,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:26,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:26,599 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,599 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:26,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:26,600 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:26,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:26,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:26,600 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:26,600 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:26,600 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:26,600 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:26,600 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:26,600 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:26,600 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:26,600 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:26,600 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:26,600 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:26,601 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:26,601 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:26,601 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:26,601 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:26,601 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,601 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:26,601 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:26,601 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,601 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:26,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:26,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:26,602 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:26,602 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:26,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:26,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:26,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:26,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:26,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:26,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@652f3544[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:26,603 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:26,603 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:26,603 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:26,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:26,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:26,604 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:26,604 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:26,604 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:26,604 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,604 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:26,604 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:26,604 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,604 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:26,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:26,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:26,604 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:26,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:26,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:26,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:26,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:26,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:26,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:26,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:26,606 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:26,606 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:26,606 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:26,606 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:26,606 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:26,607 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:26,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:26,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:26,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:26,607 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:26,608 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:26,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:26,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:26,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:26,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:26,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:26,608 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:26,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:26,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:26,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:26,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,609 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:26,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:26,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,611 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,611 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:26,611 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:26,611 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:26,611 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,611 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:26,611 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:26,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:26,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:26,611 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:26,611 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:26,611 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:26,611 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:26,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:26,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:26,611 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:26,611 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:26,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:26,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:26,712 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:26,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:26,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:26,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:26,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:26,715 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:26,715 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:26,715 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:26,715 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:26,715 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:26,715 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:26,716 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,716 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:26,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:26,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:26,716 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,716 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:26,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:26,717 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:26,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:26,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:26,717 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:26,717 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:26,717 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:26,717 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:26,717 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:26,717 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:26,717 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:26,717 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:26,717 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:26,717 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:26,717 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:26,717 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:26,717 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:26,718 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:26,718 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,718 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:26,718 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:26,718 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:26,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:26,718 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:26,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:26,719 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:26,719 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:26,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:26,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:26,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:26,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:26,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ccace28[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:26,719 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:26,719 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:26,720 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:26,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:26,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:26,720 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:26,720 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,720 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:26,720 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:26,720 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:26,721 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,721 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:26,721 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:26,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:26,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:26,721 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:26,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:26,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:26,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:26,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:26,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:26,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:26,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:26,722 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:26,722 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:26,722 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:26,722 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:26,722 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:26,723 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:26,723 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:26,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:26,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:26,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:26,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:26,723 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:26,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:26,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:26,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:26,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:26,724 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:26,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:26,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:26,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:26,724 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:26,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:26,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,726 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:26,726 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:26,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:26,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:26,726 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:26,726 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,726 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:26,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:26,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:26,726 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:26,726 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:26,726 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,726 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:26,726 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:26,726 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:26,726 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:26,726 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:26,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:26,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:26,740 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:26,740 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:26,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36277] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41978 deadline: 1681995446738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42016 deadline: 1681995446738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,740 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:26,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42008 deadline: 1681995446739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,826 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:26,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:26,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-20 12:56:26,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-20 12:56:26,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-20 12:56:26,829 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-20 12:56:26,829 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:26,829 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-20 12:56:26,829 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-20 12:56:26,829 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:26,829 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:26,830 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,830 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:26,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:26,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:26,830 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,830 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-20 12:56:26,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:26,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:26,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-20 12:56:26,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:26,831 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-20 12:56:26,831 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-20 12:56:26,831 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-20 12:56:26,831 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-20 12:56:26,831 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-20 12:56:26,831 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-20 12:56:26,832 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-20 12:56:26,832 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-20 12:56:26,832 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:26,832 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. started... 2023-04-20 12:56:26,832 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. as already flushing 2023-04-20 12:56:26,832 DEBUG [rs(jenkins-hbase20.apache.org,36277,1681995213947)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:26,832 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-20 12:56:26,832 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-20 12:56:26,832 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,832 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-20 12:56:26,832 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-20 12:56:26,833 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,833 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:26,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:26,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:26,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:26,833 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:26,833 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-20 12:56:26,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:26,834 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420a12a17ba76684161af34c68de14088bb to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420a12a17ba76684161af34c68de14088bb 2023-04-20 12:56:26,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:26,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:26,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36277,1681995213947' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-20 12:56:26,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3783ae34[Count = 0] remaining members to acquire global barrier 2023-04-20 12:56:26,836 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-20 12:56:26,836 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:26,837 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:26,837 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,837 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-20 12:56:26,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:26,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:26,837 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-20 12:56:26,837 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-20 12:56:26,837 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36277,1681995213947' in zk 2023-04-20 12:56:26,838 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=357, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/A/5f15407d350a47908fd4e03a95806077 2023-04-20 12:56:26,838 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,838 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-20 12:56:26,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:26,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:26,838 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:26,839 DEBUG [member: 'jenkins-hbase20.apache.org,36277,1681995213947' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-20 12:56:26,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:26,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:26,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:26,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:26,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:26,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36277,1681995213947': 2023-04-20 12:56:26,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36277,1681995213947' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-20 12:56:26,841 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-20 12:56:26,841 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-20 12:56:26,841 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-20 12:56:26,841 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-20 12:56:26,841 INFO [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-20 12:56:26,843 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:26,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:26,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-20 12:56:26,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-20 12:56:26,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-20 12:56:26,844 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:26,844 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:26,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:26,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:26,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-20 12:56:26,844 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:26,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:26,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:26,850 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,857 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-20 12:56:26,857 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-20 12:56:26,857 DEBUG [(jenkins-hbase20.apache.org,40401,1681995213032)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-20 12:56:26,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-20 12:56:26,857 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:26,857 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-20 12:56:26,857 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-20 12:56:26,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-20 12:56:26,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-20 12:56:26,858 DEBUG [Thread-3766] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-20 12:56:26,858 DEBUG [Thread-3766] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-20 12:56:26,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-20 12:56:26,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-20 12:56:26,857 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:26,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-20 12:56:26,858 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:26,858 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-20 12:56:26,858 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-20 12:56:26,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-20 12:56:26,859 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-20 12:56:26,861 DEBUG [Thread-3769] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x28c646e5 to 127.0.0.1:56673 2023-04-20 12:56:26,861 DEBUG [Thread-3769] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:56:26,862 DEBUG [Thread-3767] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7c686287 to 127.0.0.1:56673 2023-04-20 12:56:26,862 DEBUG [Thread-3767] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:56:26,863 DEBUG [Thread-3771] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1069406b to 127.0.0.1:56673 2023-04-20 12:56:26,863 DEBUG [Thread-3771] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:56:26,864 DEBUG [Thread-3775] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7bf3fb2a to 127.0.0.1:56673 2023-04-20 12:56:26,864 DEBUG [Thread-3775] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:56:26,867 DEBUG [Thread-3773] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1ba9b9c7 to 127.0.0.1:56673 2023-04-20 12:56:26,867 DEBUG [Thread-3773] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:56:26,958 DEBUG [Thread-3766] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-20 12:56:26,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-20 12:56:27,042 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:27,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42016 deadline: 1681995447041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:27,043 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:27,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41978 deadline: 1681995447043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:27,043 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:27,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42008 deadline: 1681995447043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:27,248 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=357 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/B/a06debfcf80144b78159ea2508cd02f6 2023-04-20 12:56:27,547 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:27,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42016 deadline: 1681995447546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:27,548 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:27,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36277] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41978 deadline: 1681995447548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:27,548 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-20 12:56:27,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42008 deadline: 1681995447548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:27,667 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=357 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/C/7b67d7177de8468cab24fbcecbae58ac 2023-04-20 12:56:27,678 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/A/5f15407d350a47908fd4e03a95806077 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/5f15407d350a47908fd4e03a95806077 2023-04-20 12:56:27,684 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/5f15407d350a47908fd4e03a95806077, entries=200, sequenceid=357, filesize=32.3 K 2023-04-20 12:56:27,685 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/B/a06debfcf80144b78159ea2508cd02f6 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/a06debfcf80144b78159ea2508cd02f6 2023-04-20 12:56:27,690 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/a06debfcf80144b78159ea2508cd02f6, entries=150, sequenceid=357, filesize=11.9 K 2023-04-20 12:56:27,691 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/C/7b67d7177de8468cab24fbcecbae58ac as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/7b67d7177de8468cab24fbcecbae58ac 2023-04-20 12:56:27,696 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/7b67d7177de8468cab24fbcecbae58ac, entries=150, sequenceid=357, filesize=11.9 K 2023-04-20 12:56:27,697 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 8113dc3e39e3420c86d60a8170c364f8 in 1273ms, sequenceid=357, compaction requested=true 2023-04-20 12:56:27,697 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8113dc3e39e3420c86d60a8170c364f8: 2023-04-20 12:56:27,697 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:56:27,697 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:56:27,697 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-20 12:56:27,697 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-20 12:56:27,697 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:56:27,699 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 130836 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-20 12:56:27,699 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1912): 8113dc3e39e3420c86d60a8170c364f8/A is initiating minor compaction (all files) 2023-04-20 12:56:27,699 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59100 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-20 12:56:27,700 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8113dc3e39e3420c86d60a8170c364f8/A in TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:27,700 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): 8113dc3e39e3420c86d60a8170c364f8/B is initiating minor compaction (all files) 2023-04-20 12:56:27,700 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/b27365ec9cea4e778caa5468f04c0e04, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/7655abce4c414d548981b78ffc8237e3, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/babf334946df4b5ebc1dc568354dfb26, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/031df4b6096c4b6483d13ceb6d43ef6b, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/5f15407d350a47908fd4e03a95806077] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp, totalSize=127.8 K 2023-04-20 12:56:27,700 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8113dc3e39e3420c86d60a8170c364f8/B in TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:27,700 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/d7e4c80585cc443a967ec8134d54d9ec, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/63d18988c9564d73a631864144a0660f, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/fca31bc4c2094bc680460ce08374c333, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/0ff9d1fb4c4f4456882e0f4285f4ff8d, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/a06debfcf80144b78159ea2508cd02f6] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp, totalSize=57.7 K 2023-04-20 12:56:27,700 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting b27365ec9cea4e778caa5468f04c0e04, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1681995380783 2023-04-20 12:56:27,700 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting d7e4c80585cc443a967ec8134d54d9ec, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1681995380783 2023-04-20 12:56:27,701 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 7655abce4c414d548981b78ffc8237e3, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1681995381927 2023-04-20 12:56:27,701 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 63d18988c9564d73a631864144a0660f, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1681995381927 2023-04-20 12:56:27,701 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting babf334946df4b5ebc1dc568354dfb26, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=324, earliestPutTs=1681995384081 2023-04-20 12:56:27,701 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting fca31bc4c2094bc680460ce08374c333, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=324, earliestPutTs=1681995384081 2023-04-20 12:56:27,701 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 031df4b6096c4b6483d13ceb6d43ef6b, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1681995384679 2023-04-20 12:56:27,702 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 0ff9d1fb4c4f4456882e0f4285f4ff8d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1681995384679 2023-04-20 12:56:27,702 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] compactions.Compactor(207): Compacting 5f15407d350a47908fd4e03a95806077, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1681995385811 2023-04-20 12:56:27,702 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting a06debfcf80144b78159ea2508cd02f6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1681995385811 2023-04-20 12:56:27,712 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): 8113dc3e39e3420c86d60a8170c364f8#B#compaction#550 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:56:27,716 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8113dc3e39e3420c86d60a8170c364f8#A#compaction#551 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:56:27,718 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/B/fb57dbbbdc5f42af9b81177eebae8e73 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/fb57dbbbdc5f42af9b81177eebae8e73 2023-04-20 12:56:27,722 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8113dc3e39e3420c86d60a8170c364f8/B of 8113dc3e39e3420c86d60a8170c364f8 into fb57dbbbdc5f42af9b81177eebae8e73(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:56:27,722 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8113dc3e39e3420c86d60a8170c364f8: 2023-04-20 12:56:27,722 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8., storeName=8113dc3e39e3420c86d60a8170c364f8/B, priority=11, startTime=1681995387697; duration=0sec 2023-04-20 12:56:27,722 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-20 12:56:27,722 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-20 12:56:27,723 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59100 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-20 12:56:27,723 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1912): 8113dc3e39e3420c86d60a8170c364f8/C is initiating minor compaction (all files) 2023-04-20 12:56:27,723 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8113dc3e39e3420c86d60a8170c364f8/C in TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:27,723 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/4a7bb375712d44ef9e765a27951db3cd, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/6ce235d43bba43668b6429741b429a69, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/7a43003ac84444dab85617dd65e1dc56, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/023026c42ddd434aa29bfb761265b1e8, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/7b67d7177de8468cab24fbcecbae58ac] into tmpdir=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp, totalSize=57.7 K 2023-04-20 12:56:27,724 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 4a7bb375712d44ef9e765a27951db3cd, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1681995380783 2023-04-20 12:56:27,724 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 6ce235d43bba43668b6429741b429a69, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1681995381927 2023-04-20 12:56:27,724 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 7a43003ac84444dab85617dd65e1dc56, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=324, earliestPutTs=1681995384081 2023-04-20 12:56:27,724 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 023026c42ddd434aa29bfb761265b1e8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1681995384679 2023-04-20 12:56:27,724 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] compactions.Compactor(207): Compacting 7b67d7177de8468cab24fbcecbae58ac, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1681995385811 2023-04-20 12:56:27,730 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] throttle.PressureAwareThroughputController(145): 8113dc3e39e3420c86d60a8170c364f8#C#compaction#552 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-20 12:56:28,138 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/C/f2f2bcb01a3942718bbc84962faf3e41 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/f2f2bcb01a3942718bbc84962faf3e41 2023-04-20 12:56:28,143 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8113dc3e39e3420c86d60a8170c364f8/C of 8113dc3e39e3420c86d60a8170c364f8 into f2f2bcb01a3942718bbc84962faf3e41(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:56:28,143 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8113dc3e39e3420c86d60a8170c364f8: 2023-04-20 12:56:28,143 INFO [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8., storeName=8113dc3e39e3420c86d60a8170c364f8/C, priority=11, startTime=1681995387697; duration=0sec 2023-04-20 12:56:28,143 DEBUG [RS:0;jenkins-hbase20:36277-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:56:28,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36277] regionserver.HRegion(9158): Flush requested on 8113dc3e39e3420c86d60a8170c364f8 2023-04-20 12:56:28,218 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8113dc3e39e3420c86d60a8170c364f8 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-20 12:56:28,218 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=A 2023-04-20 12:56:28,218 DEBUG [Thread-3762] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x67bf0d7e to 127.0.0.1:56673 2023-04-20 12:56:28,218 DEBUG [Thread-3762] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:56:28,218 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:28,219 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=B 2023-04-20 12:56:28,219 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:28,219 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=C 2023-04-20 12:56:28,219 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:28,231 DEBUG [Thread-3756] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x055c027c to 127.0.0.1:56673 2023-04-20 12:56:28,231 DEBUG [Thread-3756] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:56:28,235 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420249bbef2e62b420e944f48785756308b to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420249bbef2e62b420e944f48785756308b 2023-04-20 12:56:28,539 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/A/b59e3a2ea0c841a39b2981c9f78cfeed as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/b59e3a2ea0c841a39b2981c9f78cfeed 2023-04-20 12:56:28,546 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8113dc3e39e3420c86d60a8170c364f8/A of 8113dc3e39e3420c86d60a8170c364f8 into b59e3a2ea0c841a39b2981c9f78cfeed(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-20 12:56:28,546 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8113dc3e39e3420c86d60a8170c364f8: 2023-04-20 12:56:28,546 INFO [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8., storeName=8113dc3e39e3420c86d60a8170c364f8/A, priority=11, startTime=1681995387697; duration=0sec 2023-04-20 12:56:28,546 DEBUG [RS:0;jenkins-hbase20:36277-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-20 12:56:28,551 DEBUG [Thread-3758] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7d22e3b3 to 127.0.0.1:56673 2023-04-20 12:56:28,551 DEBUG [Thread-3758] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:56:28,552 DEBUG [Thread-3760] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x44a601bf to 127.0.0.1:56673 2023-04-20 12:56:28,552 DEBUG [Thread-3760] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:56:28,555 DEBUG [Thread-3764] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3d52316c to 127.0.0.1:56673 2023-04-20 12:56:28,555 DEBUG [Thread-3764] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:56:28,555 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-20 12:56:28,556 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(394): wrote 60 2023-04-20 12:56:28,556 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(394): wrote 64 2023-04-20 12:56:28,556 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(394): wrote 54 2023-04-20 12:56:28,556 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(394): wrote 55 2023-04-20 12:56:28,556 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(394): wrote 63 2023-04-20 12:56:28,556 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-20 12:56:28,556 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(398): read 5919 2023-04-20 12:56:28,556 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(398): read 5860 2023-04-20 12:56:28,557 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(398): read 5645 2023-04-20 12:56:28,557 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(398): read 5940 2023-04-20 12:56:28,557 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(398): read 5894 2023-04-20 12:56:28,557 INFO [Listener at localhost.localdomain/42205] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-20 12:56:28,557 INFO [Listener at localhost.localdomain/42205] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-20 12:56:28,557 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1bb3d2c9 to 127.0.0.1:56673 2023-04-20 12:56:28,557 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:56:28,558 INFO [Listener at localhost.localdomain/42205] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-20 12:56:28,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-20 12:56:28,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-20 12:56:28,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-20 12:56:28,563 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681995388563"}]},"ts":"1681995388563"} 2023-04-20 12:56:28,564 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-20 12:56:28,565 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-20 12:56:28,566 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8113dc3e39e3420c86d60a8170c364f8, UNASSIGN}] 2023-04-20 12:56:28,567 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8113dc3e39e3420c86d60a8170c364f8, UNASSIGN 2023-04-20 12:56:28,568 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=8113dc3e39e3420c86d60a8170c364f8, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:28,568 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681995388568"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1681995388568"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681995388568"}]},"ts":"1681995388568"} 2023-04-20 12:56:28,569 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure 8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947}] 2023-04-20 12:56:28,640 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=371, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/A/41cd2c18e7234064aaaef4cf0d88f8a3 2023-04-20 12:56:28,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-20 12:56:28,721 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 8113dc3e39e3420c86d60a8170c364f8 2023-04-20 12:56:28,721 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 8113dc3e39e3420c86d60a8170c364f8, disabling compactions & flushes 2023-04-20 12:56:28,721 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:28,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-20 12:56:29,049 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=371 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/B/041941c20d64440f9fd8a5ee009f05f9 2023-04-20 12:56:29,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-20 12:56:29,457 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=371 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/C/4eca3771ce8541bc99d491107d202318 2023-04-20 12:56:29,465 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/A/41cd2c18e7234064aaaef4cf0d88f8a3 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/41cd2c18e7234064aaaef4cf0d88f8a3 2023-04-20 12:56:29,474 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/41cd2c18e7234064aaaef4cf0d88f8a3, entries=150, sequenceid=371, filesize=25.4 K 2023-04-20 12:56:29,475 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/B/041941c20d64440f9fd8a5ee009f05f9 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/041941c20d64440f9fd8a5ee009f05f9 2023-04-20 12:56:29,481 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/041941c20d64440f9fd8a5ee009f05f9, entries=150, sequenceid=371, filesize=11.9 K 2023-04-20 12:56:29,482 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/C/4eca3771ce8541bc99d491107d202318 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/4eca3771ce8541bc99d491107d202318 2023-04-20 12:56:29,489 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/4eca3771ce8541bc99d491107d202318, entries=150, sequenceid=371, filesize=11.9 K 2023-04-20 12:56:29,490 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=26.84 KB/27480 for 8113dc3e39e3420c86d60a8170c364f8 in 1271ms, sequenceid=371, compaction requested=false 2023-04-20 12:56:29,490 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8113dc3e39e3420c86d60a8170c364f8: 2023-04-20 12:56:29,490 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:29,490 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:29,490 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. after waiting 0 ms 2023-04-20 12:56:29,490 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:29,490 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 8113dc3e39e3420c86d60a8170c364f8 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-20 12:56:29,490 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=A 2023-04-20 12:56:29,490 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:29,490 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=B 2023-04-20 12:56:29,491 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:29,491 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8113dc3e39e3420c86d60a8170c364f8, store=C 2023-04-20 12:56:29,491 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-20 12:56:29,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-20 12:56:29,908 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230420d1962da39e5e40a4942aaf99e0cca900 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420d1962da39e5e40a4942aaf99e0cca900 2023-04-20 12:56:29,915 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=379, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/A/7a3a13b122ed41d48c59aafafe8a4b5d 2023-04-20 12:56:29,924 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=379 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/B/135b2c3ffd704b399f0150c7c588374d 2023-04-20 12:56:30,337 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=379 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/C/c91d4b2b66e34f1d891d37e6b74bfee7 2023-04-20 12:56:30,342 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/A/7a3a13b122ed41d48c59aafafe8a4b5d as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/7a3a13b122ed41d48c59aafafe8a4b5d 2023-04-20 12:56:30,347 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/7a3a13b122ed41d48c59aafafe8a4b5d, entries=100, sequenceid=379, filesize=18.6 K 2023-04-20 12:56:30,348 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/B/135b2c3ffd704b399f0150c7c588374d as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/135b2c3ffd704b399f0150c7c588374d 2023-04-20 12:56:30,352 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/135b2c3ffd704b399f0150c7c588374d, entries=100, sequenceid=379, filesize=9.5 K 2023-04-20 12:56:30,352 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/.tmp/C/c91d4b2b66e34f1d891d37e6b74bfee7 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/c91d4b2b66e34f1d891d37e6b74bfee7 2023-04-20 12:56:30,356 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/c91d4b2b66e34f1d891d37e6b74bfee7, entries=100, sequenceid=379, filesize=9.5 K 2023-04-20 12:56:30,356 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 8113dc3e39e3420c86d60a8170c364f8 in 866ms, sequenceid=379, compaction requested=true 2023-04-20 12:56:30,375 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/8db65877342f457990d6b47df3d2be63, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/5121c5cbff4447f187104c5598bded90, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/a7107293519240149a6f840406885f2e, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/00b4802d234046e894dc99af2e9e491d, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/d07c24c693cb4f71a50db907611b70a9, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/6c4802930e584eb1bf31e4b8f091bdb8, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/ef945febbc434b91ba3d74223e35c169, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/bef4c7402e3945f589b20ce014cc9974, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/ab12c5e34e064f1c8504d5fe878add97, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/475193737fc24ed789fc466efe4b52bd, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/52f3560285f044ceb2059bf69522f0c9, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/f6c0276a42f14a699bd6cbe62d202a85, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/ed61dc61ec2e4471a8dfd8fadaf8c35e, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/56f748456cc046f1b054c6753401e26f, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/25a218c148c6407b9b153833c1f692c1, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/31e501f670fb43ce9795cef57068b463, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/8b128815343541178677c7e204c81111, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/cdc3c7dcae39452e960a8cddfeaffb7a, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/b27365ec9cea4e778caa5468f04c0e04, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/7655abce4c414d548981b78ffc8237e3, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/babf334946df4b5ebc1dc568354dfb26, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/031df4b6096c4b6483d13ceb6d43ef6b, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/5f15407d350a47908fd4e03a95806077] to archive 2023-04-20 12:56:30,375 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-20 12:56:30,377 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/8db65877342f457990d6b47df3d2be63 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/8db65877342f457990d6b47df3d2be63 2023-04-20 12:56:30,378 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/5121c5cbff4447f187104c5598bded90 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/5121c5cbff4447f187104c5598bded90 2023-04-20 12:56:30,379 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/a7107293519240149a6f840406885f2e to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/a7107293519240149a6f840406885f2e 2023-04-20 12:56:30,380 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/00b4802d234046e894dc99af2e9e491d to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/00b4802d234046e894dc99af2e9e491d 2023-04-20 12:56:30,381 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/d07c24c693cb4f71a50db907611b70a9 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/d07c24c693cb4f71a50db907611b70a9 2023-04-20 12:56:30,382 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/6c4802930e584eb1bf31e4b8f091bdb8 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/6c4802930e584eb1bf31e4b8f091bdb8 2023-04-20 12:56:30,383 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/ef945febbc434b91ba3d74223e35c169 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/ef945febbc434b91ba3d74223e35c169 2023-04-20 12:56:30,384 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/bef4c7402e3945f589b20ce014cc9974 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/bef4c7402e3945f589b20ce014cc9974 2023-04-20 12:56:30,385 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/ab12c5e34e064f1c8504d5fe878add97 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/ab12c5e34e064f1c8504d5fe878add97 2023-04-20 12:56:30,386 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/475193737fc24ed789fc466efe4b52bd to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/475193737fc24ed789fc466efe4b52bd 2023-04-20 12:56:30,386 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/52f3560285f044ceb2059bf69522f0c9 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/52f3560285f044ceb2059bf69522f0c9 2023-04-20 12:56:30,387 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/f6c0276a42f14a699bd6cbe62d202a85 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/f6c0276a42f14a699bd6cbe62d202a85 2023-04-20 12:56:30,388 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/ed61dc61ec2e4471a8dfd8fadaf8c35e to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/ed61dc61ec2e4471a8dfd8fadaf8c35e 2023-04-20 12:56:30,389 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/56f748456cc046f1b054c6753401e26f to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/56f748456cc046f1b054c6753401e26f 2023-04-20 12:56:30,390 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/25a218c148c6407b9b153833c1f692c1 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/25a218c148c6407b9b153833c1f692c1 2023-04-20 12:56:30,391 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/31e501f670fb43ce9795cef57068b463 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/31e501f670fb43ce9795cef57068b463 2023-04-20 12:56:30,392 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/8b128815343541178677c7e204c81111 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/8b128815343541178677c7e204c81111 2023-04-20 12:56:30,393 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/cdc3c7dcae39452e960a8cddfeaffb7a to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/cdc3c7dcae39452e960a8cddfeaffb7a 2023-04-20 12:56:30,394 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/b27365ec9cea4e778caa5468f04c0e04 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/b27365ec9cea4e778caa5468f04c0e04 2023-04-20 12:56:30,395 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/7655abce4c414d548981b78ffc8237e3 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/7655abce4c414d548981b78ffc8237e3 2023-04-20 12:56:30,396 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/babf334946df4b5ebc1dc568354dfb26 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/babf334946df4b5ebc1dc568354dfb26 2023-04-20 12:56:30,397 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/031df4b6096c4b6483d13ceb6d43ef6b to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/031df4b6096c4b6483d13ceb6d43ef6b 2023-04-20 12:56:30,398 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/5f15407d350a47908fd4e03a95806077 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/5f15407d350a47908fd4e03a95806077 2023-04-20 12:56:30,401 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/a8b74ab8168d4f33bd1b3a2518d39221, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/9bc0ae0820244d6a8a1947109cc06c70, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/c2a8189a8fee4afbaa5980026c54d782, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/c6e30f105f2540e3a16aab5c3db2eeb4, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/f04ab1daa88445b9a38f487cc3fe6725, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/3252163496a3495cae7662d0aa46ab38, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/5ddb5df9bd9c4ecdba8287d3ee2cbfca, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/a381f1c4edda40559141761107fd7e01, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/d06e6a61e9444b7daf40d0040496eb54, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/015c558d946e464189962e779e9fa3a6, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/cbc4d02c68ed4708ae7e7e09ff150dc8, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/e9aa6376c3de4fd6b747f739fa0b5dfc, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/88803527119348b4ada46bd7f6e7cd18, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/3578e057e5e34551a249bf5d9f59fc6d, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/a1f7f6b3890c470b98e2b304f9c4dd21, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/a318c92e343d48f9b09a4f791653b6c6, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/4df6e7895f434ffdad8ecad0b8a138b9, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/d7e4c80585cc443a967ec8134d54d9ec, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/c987e8c0f81c44dd84258d02a5034b2c, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/63d18988c9564d73a631864144a0660f, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/fca31bc4c2094bc680460ce08374c333, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/0ff9d1fb4c4f4456882e0f4285f4ff8d, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/a06debfcf80144b78159ea2508cd02f6] to archive 2023-04-20 12:56:30,402 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-20 12:56:30,403 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/a8b74ab8168d4f33bd1b3a2518d39221 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/a8b74ab8168d4f33bd1b3a2518d39221 2023-04-20 12:56:30,404 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/9bc0ae0820244d6a8a1947109cc06c70 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/9bc0ae0820244d6a8a1947109cc06c70 2023-04-20 12:56:30,405 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/c2a8189a8fee4afbaa5980026c54d782 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/c2a8189a8fee4afbaa5980026c54d782 2023-04-20 12:56:30,406 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/c6e30f105f2540e3a16aab5c3db2eeb4 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/c6e30f105f2540e3a16aab5c3db2eeb4 2023-04-20 12:56:30,408 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/f04ab1daa88445b9a38f487cc3fe6725 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/f04ab1daa88445b9a38f487cc3fe6725 2023-04-20 12:56:30,409 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/3252163496a3495cae7662d0aa46ab38 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/3252163496a3495cae7662d0aa46ab38 2023-04-20 12:56:30,410 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/5ddb5df9bd9c4ecdba8287d3ee2cbfca to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/5ddb5df9bd9c4ecdba8287d3ee2cbfca 2023-04-20 12:56:30,411 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/a381f1c4edda40559141761107fd7e01 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/a381f1c4edda40559141761107fd7e01 2023-04-20 12:56:30,412 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/d06e6a61e9444b7daf40d0040496eb54 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/d06e6a61e9444b7daf40d0040496eb54 2023-04-20 12:56:30,413 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/015c558d946e464189962e779e9fa3a6 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/015c558d946e464189962e779e9fa3a6 2023-04-20 12:56:30,414 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/cbc4d02c68ed4708ae7e7e09ff150dc8 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/cbc4d02c68ed4708ae7e7e09ff150dc8 2023-04-20 12:56:30,415 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/e9aa6376c3de4fd6b747f739fa0b5dfc to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/e9aa6376c3de4fd6b747f739fa0b5dfc 2023-04-20 12:56:30,417 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/88803527119348b4ada46bd7f6e7cd18 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/88803527119348b4ada46bd7f6e7cd18 2023-04-20 12:56:30,418 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/3578e057e5e34551a249bf5d9f59fc6d to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/3578e057e5e34551a249bf5d9f59fc6d 2023-04-20 12:56:30,419 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/a1f7f6b3890c470b98e2b304f9c4dd21 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/a1f7f6b3890c470b98e2b304f9c4dd21 2023-04-20 12:56:30,421 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/a318c92e343d48f9b09a4f791653b6c6 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/a318c92e343d48f9b09a4f791653b6c6 2023-04-20 12:56:30,422 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/4df6e7895f434ffdad8ecad0b8a138b9 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/4df6e7895f434ffdad8ecad0b8a138b9 2023-04-20 12:56:30,424 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/d7e4c80585cc443a967ec8134d54d9ec to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/d7e4c80585cc443a967ec8134d54d9ec 2023-04-20 12:56:30,425 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/c987e8c0f81c44dd84258d02a5034b2c to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/c987e8c0f81c44dd84258d02a5034b2c 2023-04-20 12:56:30,426 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/63d18988c9564d73a631864144a0660f to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/63d18988c9564d73a631864144a0660f 2023-04-20 12:56:30,428 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/fca31bc4c2094bc680460ce08374c333 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/fca31bc4c2094bc680460ce08374c333 2023-04-20 12:56:30,430 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/0ff9d1fb4c4f4456882e0f4285f4ff8d to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/0ff9d1fb4c4f4456882e0f4285f4ff8d 2023-04-20 12:56:30,431 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/a06debfcf80144b78159ea2508cd02f6 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/a06debfcf80144b78159ea2508cd02f6 2023-04-20 12:56:30,435 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/5d85ed4b5a7b4cf4916ec2c87c7c1d83, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/fd4478ca22164caa9eecaa98a945d3d0, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/0c8e6efed97a44658ef7188ab811de74, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/ac33a56c447f4806bab626b20fdcf70f, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/2e46722a70684cecb1d46eccdb40343d, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/89a20537a1824ef7b5c4fd5cc79e351a, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/8041c0eb51e249009ed8d3e439bbc2b2, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/3b95b18cd7194c839aa240eae38ed958, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/9990be75bc4241e0adbfbe1feedf6323, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/6bd9cd0da872410fb37968229f701cc8, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/2f4a9d359c4d429091ad9bf8373a4ff1, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/9913a77d6b1d424eb22bc682cb7290f7, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/c364e0615c134c7a8e30f977d2eee77a, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/13e1c545f4db47329cb58643a62e5f81, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/e1fcb8fccdd64dc1a626c87191d8f7b0, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/0a9c01fcbaf640f1b129e139d6cc7268, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/7e7129915fa040cca85351312ac89940, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/4a7bb375712d44ef9e765a27951db3cd, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/4b53723f1d794214b29548db1e5ee2fa, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/6ce235d43bba43668b6429741b429a69, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/7a43003ac84444dab85617dd65e1dc56, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/023026c42ddd434aa29bfb761265b1e8, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/7b67d7177de8468cab24fbcecbae58ac] to archive 2023-04-20 12:56:30,436 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-20 12:56:30,438 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/5d85ed4b5a7b4cf4916ec2c87c7c1d83 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/5d85ed4b5a7b4cf4916ec2c87c7c1d83 2023-04-20 12:56:30,440 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/fd4478ca22164caa9eecaa98a945d3d0 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/fd4478ca22164caa9eecaa98a945d3d0 2023-04-20 12:56:30,442 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/0c8e6efed97a44658ef7188ab811de74 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/0c8e6efed97a44658ef7188ab811de74 2023-04-20 12:56:30,444 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/ac33a56c447f4806bab626b20fdcf70f to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/ac33a56c447f4806bab626b20fdcf70f 2023-04-20 12:56:30,445 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/2e46722a70684cecb1d46eccdb40343d to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/2e46722a70684cecb1d46eccdb40343d 2023-04-20 12:56:30,447 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/89a20537a1824ef7b5c4fd5cc79e351a to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/89a20537a1824ef7b5c4fd5cc79e351a 2023-04-20 12:56:30,449 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/8041c0eb51e249009ed8d3e439bbc2b2 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/8041c0eb51e249009ed8d3e439bbc2b2 2023-04-20 12:56:30,450 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/3b95b18cd7194c839aa240eae38ed958 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/3b95b18cd7194c839aa240eae38ed958 2023-04-20 12:56:30,452 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/9990be75bc4241e0adbfbe1feedf6323 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/9990be75bc4241e0adbfbe1feedf6323 2023-04-20 12:56:30,454 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/6bd9cd0da872410fb37968229f701cc8 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/6bd9cd0da872410fb37968229f701cc8 2023-04-20 12:56:30,456 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/2f4a9d359c4d429091ad9bf8373a4ff1 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/2f4a9d359c4d429091ad9bf8373a4ff1 2023-04-20 12:56:30,458 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/9913a77d6b1d424eb22bc682cb7290f7 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/9913a77d6b1d424eb22bc682cb7290f7 2023-04-20 12:56:30,460 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/c364e0615c134c7a8e30f977d2eee77a to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/c364e0615c134c7a8e30f977d2eee77a 2023-04-20 12:56:30,462 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/13e1c545f4db47329cb58643a62e5f81 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/13e1c545f4db47329cb58643a62e5f81 2023-04-20 12:56:30,464 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/e1fcb8fccdd64dc1a626c87191d8f7b0 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/e1fcb8fccdd64dc1a626c87191d8f7b0 2023-04-20 12:56:30,465 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/0a9c01fcbaf640f1b129e139d6cc7268 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/0a9c01fcbaf640f1b129e139d6cc7268 2023-04-20 12:56:30,467 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/7e7129915fa040cca85351312ac89940 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/7e7129915fa040cca85351312ac89940 2023-04-20 12:56:30,469 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/4a7bb375712d44ef9e765a27951db3cd to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/4a7bb375712d44ef9e765a27951db3cd 2023-04-20 12:56:30,471 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/4b53723f1d794214b29548db1e5ee2fa to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/4b53723f1d794214b29548db1e5ee2fa 2023-04-20 12:56:30,473 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/6ce235d43bba43668b6429741b429a69 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/6ce235d43bba43668b6429741b429a69 2023-04-20 12:56:30,475 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/7a43003ac84444dab85617dd65e1dc56 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/7a43003ac84444dab85617dd65e1dc56 2023-04-20 12:56:30,476 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/023026c42ddd434aa29bfb761265b1e8 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/023026c42ddd434aa29bfb761265b1e8 2023-04-20 12:56:30,478 DEBUG [StoreCloser-TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/7b67d7177de8468cab24fbcecbae58ac to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/7b67d7177de8468cab24fbcecbae58ac 2023-04-20 12:56:30,483 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/recovered.edits/382.seqid, newMaxSeqId=382, maxSeqId=4 2023-04-20 12:56:30,484 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8. 2023-04-20 12:56:30,485 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 8113dc3e39e3420c86d60a8170c364f8: 2023-04-20 12:56:30,487 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 8113dc3e39e3420c86d60a8170c364f8 2023-04-20 12:56:30,488 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=8113dc3e39e3420c86d60a8170c364f8, regionState=CLOSED 2023-04-20 12:56:30,488 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681995390488"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681995390488"}]},"ts":"1681995390488"} 2023-04-20 12:56:30,491 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-04-20 12:56:30,491 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure 8113dc3e39e3420c86d60a8170c364f8, server=jenkins-hbase20.apache.org,36277,1681995213947 in 1.9200 sec 2023-04-20 12:56:30,494 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-04-20 12:56:30,494 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=8113dc3e39e3420c86d60a8170c364f8, UNASSIGN in 1.9250 sec 2023-04-20 12:56:30,494 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681995390494"}]},"ts":"1681995390494"} 2023-04-20 12:56:30,496 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-20 12:56:30,497 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-20 12:56:30,499 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9380 sec 2023-04-20 12:56:30,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-20 12:56:30,672 INFO [Listener at localhost.localdomain/42205] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-04-20 12:56:30,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-20 12:56:30,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 12:56:30,680 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 12:56:30,681 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 12:56:30,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-20 12:56:30,690 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8 2023-04-20 12:56:30,694 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A, FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B, FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C, FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/recovered.edits] 2023-04-20 12:56:30,697 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/41cd2c18e7234064aaaef4cf0d88f8a3 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/41cd2c18e7234064aaaef4cf0d88f8a3 2023-04-20 12:56:30,699 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/7a3a13b122ed41d48c59aafafe8a4b5d to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/7a3a13b122ed41d48c59aafafe8a4b5d 2023-04-20 12:56:30,700 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/b59e3a2ea0c841a39b2981c9f78cfeed to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/A/b59e3a2ea0c841a39b2981c9f78cfeed 2023-04-20 12:56:30,702 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/041941c20d64440f9fd8a5ee009f05f9 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/041941c20d64440f9fd8a5ee009f05f9 2023-04-20 12:56:30,704 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/135b2c3ffd704b399f0150c7c588374d to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/135b2c3ffd704b399f0150c7c588374d 2023-04-20 12:56:30,705 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/fb57dbbbdc5f42af9b81177eebae8e73 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/B/fb57dbbbdc5f42af9b81177eebae8e73 2023-04-20 12:56:30,707 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/4eca3771ce8541bc99d491107d202318 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/4eca3771ce8541bc99d491107d202318 2023-04-20 12:56:30,708 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/c91d4b2b66e34f1d891d37e6b74bfee7 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/c91d4b2b66e34f1d891d37e6b74bfee7 2023-04-20 12:56:30,709 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/f2f2bcb01a3942718bbc84962faf3e41 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/C/f2f2bcb01a3942718bbc84962faf3e41 2023-04-20 12:56:30,711 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/recovered.edits/382.seqid to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8/recovered.edits/382.seqid 2023-04-20 12:56:30,712 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/.tmp/data/default/TestAcidGuarantees/8113dc3e39e3420c86d60a8170c364f8 2023-04-20 12:56:30,712 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-20 12:56:30,712 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-20 12:56:30,713 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-20 12:56:30,715 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304200729156cfe6f46428f39fe76cfa7a3dc to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304200729156cfe6f46428f39fe76cfa7a3dc 2023-04-20 12:56:30,716 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304200bfa43cbf72b49bd8c7ce87cc4f90208 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304200bfa43cbf72b49bd8c7ce87cc4f90208 2023-04-20 12:56:30,717 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304201b0b5bd137814441b3bfcfd021299364 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304201b0b5bd137814441b3bfcfd021299364 2023-04-20 12:56:30,718 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304201e06d8a22ce64c8c86b877089259001b to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304201e06d8a22ce64c8c86b877089259001b 2023-04-20 12:56:30,719 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420249bbef2e62b420e944f48785756308b to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420249bbef2e62b420e944f48785756308b 2023-04-20 12:56:30,721 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304202c4399a7dc8547bda4bdd823f7ca2e5d to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304202c4399a7dc8547bda4bdd823f7ca2e5d 2023-04-20 12:56:30,722 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042039efa8df50834757b0d07003ad8ca949 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023042039efa8df50834757b0d07003ad8ca949 2023-04-20 12:56:30,723 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420760b394812d941928c4224da5ff0a6e5 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420760b394812d941928c4224da5ff0a6e5 2023-04-20 12:56:30,725 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304207da0196c58694ae9a26388df7db0900c to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304207da0196c58694ae9a26388df7db0900c 2023-04-20 12:56:30,726 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304209670b44cccf648d58a2ab9baf5dcc48e to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304209670b44cccf648d58a2ab9baf5dcc48e 2023-04-20 12:56:30,728 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420997f9918d554485c816fad6bfd25dfc8 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420997f9918d554485c816fad6bfd25dfc8 2023-04-20 12:56:30,730 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420a12a17ba76684161af34c68de14088bb to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420a12a17ba76684161af34c68de14088bb 2023-04-20 12:56:30,732 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420b052abf9e09542d3bd94148b71708ad7 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420b052abf9e09542d3bd94148b71708ad7 2023-04-20 12:56:30,734 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420b7ea28ffbcb1438fb92aa2cd67b77bbb to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420b7ea28ffbcb1438fb92aa2cd67b77bbb 2023-04-20 12:56:30,736 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420c217beb2ead14193971976ce00af00d5 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420c217beb2ead14193971976ce00af00d5 2023-04-20 12:56:30,738 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420d163f7bbb3084794806ee281cd4ff788 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420d163f7bbb3084794806ee281cd4ff788 2023-04-20 12:56:30,741 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420d1962da39e5e40a4942aaf99e0cca900 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420d1962da39e5e40a4942aaf99e0cca900 2023-04-20 12:56:30,743 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420d422bc54fcde4d8891a55ad80473aaf0 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420d422bc54fcde4d8891a55ad80473aaf0 2023-04-20 12:56:30,744 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420f013a9fc63d046489272b3332a838668 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420f013a9fc63d046489272b3332a838668 2023-04-20 12:56:30,746 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420fc27da1762984c2fa411ca67892582ce to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420fc27da1762984c2fa411ca67892582ce 2023-04-20 12:56:30,748 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420ff11f43851ee43629f37ec89f4fd4cfc to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420ff11f43851ee43629f37ec89f4fd4cfc 2023-04-20 12:56:30,750 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420ff71795432f74a24b9e6a6af0a1dc633 to hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230420ff71795432f74a24b9e6a6af0a1dc633 2023-04-20 12:56:30,750 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-20 12:56:30,753 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 12:56:30,755 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-20 12:56:30,757 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-20 12:56:30,758 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 12:56:30,758 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-20 12:56:30,759 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681995390759"}]},"ts":"9223372036854775807"} 2023-04-20 12:56:30,761 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-20 12:56:30,761 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 8113dc3e39e3420c86d60a8170c364f8, NAME => 'TestAcidGuarantees,,1681995363733.8113dc3e39e3420c86d60a8170c364f8.', STARTKEY => '', ENDKEY => ''}] 2023-04-20 12:56:30,761 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-20 12:56:30,761 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681995390761"}]},"ts":"9223372036854775807"} 2023-04-20 12:56:30,764 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-20 12:56:30,766 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-20 12:56:30,768 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 91 msec 2023-04-20 12:56:30,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40401] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-20 12:56:30,783 INFO [Listener at localhost.localdomain/42205] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-04-20 12:56:30,799 INFO [Listener at localhost.localdomain/42205] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=230 (was 232), OpenFileDescriptor=540 (was 552), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=358 (was 346) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 167), AvailableMemoryMB=9447 (was 9448) 2023-04-20 12:56:30,800 INFO [Listener at localhost.localdomain/42205] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-04-20 12:56:30,800 INFO [Listener at localhost.localdomain/42205] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-20 12:56:30,800 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4c9bc423 to 127.0.0.1:56673 2023-04-20 12:56:30,801 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:56:30,801 DEBUG [Listener at localhost.localdomain/42205] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-04-20 12:56:30,801 DEBUG [Listener at localhost.localdomain/42205] util.JVMClusterUtil(257): Found active master hash=1153174393, stopped=false 2023-04-20 12:56:30,801 INFO [Listener at localhost.localdomain/42205] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase20.apache.org,40401,1681995213032 2023-04-20 12:56:30,802 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-20 12:56:30,802 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-20 12:56:30,802 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-20 12:56:30,802 INFO [Listener at localhost.localdomain/42205] procedure2.ProcedureExecutor(629): Stopping 2023-04-20 12:56:30,802 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-20 12:56:30,802 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-20 12:56:30,802 DEBUG [Listener at localhost.localdomain/42205] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x38302e48 to 127.0.0.1:56673 2023-04-20 12:56:30,802 DEBUG [Listener at localhost.localdomain/42205] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:56:30,803 INFO [Listener at localhost.localdomain/42205] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase20.apache.org,36277,1681995213947' ***** 2023-04-20 12:56:30,803 INFO [Listener at localhost.localdomain/42205] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-20 12:56:30,803 INFO [RS:0;jenkins-hbase20:36277] regionserver.HeapMemoryManager(220): Stopping 2023-04-20 12:56:30,803 INFO [RS:0;jenkins-hbase20:36277] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-20 12:56:30,803 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-20 12:56:30,803 INFO [RS:0;jenkins-hbase20:36277] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-20 12:56:30,803 INFO [RS:0;jenkins-hbase20:36277] regionserver.HRegionServer(3304): Received CLOSE for 4b9689ab36fe7756d2951e73df110950 2023-04-20 12:56:30,804 INFO [RS:0;jenkins-hbase20:36277] regionserver.HRegionServer(1145): stopping server jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:30,804 DEBUG [RS:0;jenkins-hbase20:36277] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2a9f12d4 to 127.0.0.1:56673 2023-04-20 12:56:30,804 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 4b9689ab36fe7756d2951e73df110950, disabling compactions & flushes 2023-04-20 12:56:30,804 DEBUG [RS:0;jenkins-hbase20:36277] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:56:30,804 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1681995215982.4b9689ab36fe7756d2951e73df110950. 2023-04-20 12:56:30,804 INFO [RS:0;jenkins-hbase20:36277] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-20 12:56:30,804 INFO [RS:0;jenkins-hbase20:36277] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-20 12:56:30,805 INFO [RS:0;jenkins-hbase20:36277] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-20 12:56:30,804 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1681995215982.4b9689ab36fe7756d2951e73df110950. 2023-04-20 12:56:30,805 INFO [RS:0;jenkins-hbase20:36277] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-04-20 12:56:30,805 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1681995215982.4b9689ab36fe7756d2951e73df110950. after waiting 0 ms 2023-04-20 12:56:30,805 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1681995215982.4b9689ab36fe7756d2951e73df110950. 2023-04-20 12:56:30,805 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 4b9689ab36fe7756d2951e73df110950 1/1 column families, dataSize=78 B heapSize=488 B 2023-04-20 12:56:30,805 INFO [RS:0;jenkins-hbase20:36277] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-04-20 12:56:30,805 DEBUG [RS:0;jenkins-hbase20:36277] regionserver.HRegionServer(1479): Online Regions={4b9689ab36fe7756d2951e73df110950=hbase:namespace,,1681995215982.4b9689ab36fe7756d2951e73df110950., 1588230740=hbase:meta,,1.1588230740} 2023-04-20 12:56:30,805 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-20 12:56:30,805 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-20 12:56:30,805 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-20 12:56:30,805 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-20 12:56:30,805 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-20 12:56:30,805 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.94 KB heapSize=36.31 KB 2023-04-20 12:56:30,807 DEBUG [RS:0;jenkins-hbase20:36277] regionserver.HRegionServer(1505): Waiting on 1588230740, 4b9689ab36fe7756d2951e73df110950 2023-04-20 12:56:30,816 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.29 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/hbase/meta/1588230740/.tmp/info/58bf694b12f447f994d17a764da2be6b 2023-04-20 12:56:30,821 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 58bf694b12f447f994d17a764da2be6b 2023-04-20 12:56:30,850 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-20 12:56:31,007 DEBUG [RS:0;jenkins-hbase20:36277] regionserver.HRegionServer(1505): Waiting on 1588230740, 4b9689ab36fe7756d2951e73df110950 2023-04-20 12:56:31,108 INFO [regionserver/jenkins-hbase20:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-04-20 12:56:31,108 INFO [regionserver/jenkins-hbase20:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-04-20 12:56:31,208 DEBUG [RS:0;jenkins-hbase20:36277] regionserver.HRegionServer(1505): Waiting on 1588230740, 4b9689ab36fe7756d2951e73df110950 2023-04-20 12:56:31,216 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/hbase/namespace/4b9689ab36fe7756d2951e73df110950/.tmp/info/8fa4b0f8de864b4ab254388697b7d97d 2023-04-20 12:56:31,224 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/hbase/namespace/4b9689ab36fe7756d2951e73df110950/.tmp/info/8fa4b0f8de864b4ab254388697b7d97d as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/hbase/namespace/4b9689ab36fe7756d2951e73df110950/info/8fa4b0f8de864b4ab254388697b7d97d 2023-04-20 12:56:31,226 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/hbase/meta/1588230740/.tmp/rep_barrier/218826c3f4884b7c96bf808b35c0737f 2023-04-20 12:56:31,229 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/hbase/namespace/4b9689ab36fe7756d2951e73df110950/info/8fa4b0f8de864b4ab254388697b7d97d, entries=2, sequenceid=6, filesize=4.8 K 2023-04-20 12:56:31,230 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 4b9689ab36fe7756d2951e73df110950 in 425ms, sequenceid=6, compaction requested=false 2023-04-20 12:56:31,231 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 218826c3f4884b7c96bf808b35c0737f 2023-04-20 12:56:31,235 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/hbase/namespace/4b9689ab36fe7756d2951e73df110950/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-04-20 12:56:31,236 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1681995215982.4b9689ab36fe7756d2951e73df110950. 2023-04-20 12:56:31,236 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 4b9689ab36fe7756d2951e73df110950: 2023-04-20 12:56:31,236 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1681995215982.4b9689ab36fe7756d2951e73df110950. 2023-04-20 12:56:31,239 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/hbase/meta/1588230740/.tmp/table/43e6199032f14ed2a6dd103732cb7dcf 2023-04-20 12:56:31,243 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 43e6199032f14ed2a6dd103732cb7dcf 2023-04-20 12:56:31,244 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/hbase/meta/1588230740/.tmp/info/58bf694b12f447f994d17a764da2be6b as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/hbase/meta/1588230740/info/58bf694b12f447f994d17a764da2be6b 2023-04-20 12:56:31,248 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 58bf694b12f447f994d17a764da2be6b 2023-04-20 12:56:31,248 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/hbase/meta/1588230740/info/58bf694b12f447f994d17a764da2be6b, entries=22, sequenceid=93, filesize=7.1 K 2023-04-20 12:56:31,249 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/hbase/meta/1588230740/.tmp/rep_barrier/218826c3f4884b7c96bf808b35c0737f as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/hbase/meta/1588230740/rep_barrier/218826c3f4884b7c96bf808b35c0737f 2023-04-20 12:56:31,253 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 218826c3f4884b7c96bf808b35c0737f 2023-04-20 12:56:31,254 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/hbase/meta/1588230740/rep_barrier/218826c3f4884b7c96bf808b35c0737f, entries=6, sequenceid=93, filesize=5.5 K 2023-04-20 12:56:31,254 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/hbase/meta/1588230740/.tmp/table/43e6199032f14ed2a6dd103732cb7dcf as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/hbase/meta/1588230740/table/43e6199032f14ed2a6dd103732cb7dcf 2023-04-20 12:56:31,259 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 43e6199032f14ed2a6dd103732cb7dcf 2023-04-20 12:56:31,259 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/hbase/meta/1588230740/table/43e6199032f14ed2a6dd103732cb7dcf, entries=9, sequenceid=93, filesize=5.5 K 2023-04-20 12:56:31,259 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.94 KB/21446, heapSize ~36.27 KB/37136, currentSize=0 B/0 for 1588230740 in 454ms, sequenceid=93, compaction requested=false 2023-04-20 12:56:31,271 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-04-20 12:56:31,272 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-20 12:56:31,273 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-20 12:56:31,274 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-20 12:56:31,274 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-04-20 12:56:31,408 INFO [RS:0;jenkins-hbase20:36277] regionserver.HRegionServer(1171): stopping server jenkins-hbase20.apache.org,36277,1681995213947; all regions closed. 2023-04-20 12:56:31,417 DEBUG [RS:0;jenkins-hbase20:36277] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/oldWALs 2023-04-20 12:56:31,417 INFO [RS:0;jenkins-hbase20:36277] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase20.apache.org%2C36277%2C1681995213947.meta:.meta(num 1681995215748) 2023-04-20 12:56:31,422 DEBUG [RS:0;jenkins-hbase20:36277] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/oldWALs 2023-04-20 12:56:31,422 INFO [RS:0;jenkins-hbase20:36277] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase20.apache.org%2C36277%2C1681995213947:(num 1681995215254) 2023-04-20 12:56:31,422 DEBUG [RS:0;jenkins-hbase20:36277] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:56:31,422 INFO [RS:0;jenkins-hbase20:36277] regionserver.LeaseManager(133): Closed leases 2023-04-20 12:56:31,422 INFO [RS:0;jenkins-hbase20:36277] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase20:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-04-20 12:56:31,422 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-20 12:56:31,424 INFO [RS:0;jenkins-hbase20:36277] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:36277 2023-04-20 12:56:31,428 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-20 12:56:31,428 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,36277,1681995213947 2023-04-20 12:56:31,429 ERROR [Listener at localhost.localdomain/42205-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5d5d396d rejected from java.util.concurrent.ThreadPoolExecutor@5962b61e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2678] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-20 12:56:31,429 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-20 12:56:31,429 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase20.apache.org,36277,1681995213947] 2023-04-20 12:56:31,429 ERROR [Listener at localhost.localdomain/42205-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1914ec19 rejected from java.util.concurrent.ThreadPoolExecutor@5962b61e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2678] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-20 12:56:31,430 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase20.apache.org,36277,1681995213947; numProcessing=1 2023-04-20 12:56:31,430 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase20.apache.org,36277,1681995213947 already deleted, retry=false 2023-04-20 12:56:31,430 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase20.apache.org,36277,1681995213947 expired; onlineServers=0 2023-04-20 12:56:31,431 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase20.apache.org,40401,1681995213032' ***** 2023-04-20 12:56:31,431 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-04-20 12:56:31,431 DEBUG [M:0;jenkins-hbase20:40401] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5810142, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-04-20 12:56:31,431 INFO [M:0;jenkins-hbase20:40401] regionserver.HRegionServer(1145): stopping server jenkins-hbase20.apache.org,40401,1681995213032 2023-04-20 12:56:31,431 INFO [M:0;jenkins-hbase20:40401] regionserver.HRegionServer(1171): stopping server jenkins-hbase20.apache.org,40401,1681995213032; all regions closed. 2023-04-20 12:56:31,431 DEBUG [M:0;jenkins-hbase20:40401] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-20 12:56:31,431 DEBUG [M:0;jenkins-hbase20:40401] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-04-20 12:56:31,431 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-04-20 12:56:31,431 DEBUG [M:0;jenkins-hbase20:40401] cleaner.HFileCleaner(317): Stopping file delete threads 2023-04-20 12:56:31,431 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1681995214901] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1681995214901,5,FailOnTimeoutGroup] 2023-04-20 12:56:31,431 INFO [M:0;jenkins-hbase20:40401] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-04-20 12:56:31,431 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1681995214901] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1681995214901,5,FailOnTimeoutGroup] 2023-04-20 12:56:31,431 INFO [M:0;jenkins-hbase20:40401] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-04-20 12:56:31,432 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-04-20 12:56:31,432 INFO [M:0;jenkins-hbase20:40401] hbase.ChoreService(369): Chore service for: master/jenkins-hbase20:0 had [] on shutdown 2023-04-20 12:56:31,432 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-20 12:56:31,433 DEBUG [M:0;jenkins-hbase20:40401] master.HMaster(1512): Stopping service threads 2023-04-20 12:56:31,433 INFO [M:0;jenkins-hbase20:40401] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-04-20 12:56:31,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-20 12:56:31,433 ERROR [M:0;jenkins-hbase20:40401] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-11,5,PEWorkerGroup] Thread[HFileArchiver-12,5,PEWorkerGroup] Thread[IPC Client (1561327437) connection to localhost.localdomain/127.0.0.1:45293 from jenkins,5,PEWorkerGroup] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-04-20 12:56:31,434 INFO [M:0;jenkins-hbase20:40401] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-20 12:56:31,434 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-04-20 12:56:31,434 DEBUG [M:0;jenkins-hbase20:40401] zookeeper.ZKUtil(398): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-04-20 12:56:31,434 WARN [M:0;jenkins-hbase20:40401] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-04-20 12:56:31,434 INFO [M:0;jenkins-hbase20:40401] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-20 12:56:31,434 INFO [M:0;jenkins-hbase20:40401] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-04-20 12:56:31,435 DEBUG [M:0;jenkins-hbase20:40401] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-20 12:56:31,435 INFO [M:0;jenkins-hbase20:40401] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-20 12:56:31,435 DEBUG [M:0;jenkins-hbase20:40401] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-20 12:56:31,435 DEBUG [M:0;jenkins-hbase20:40401] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-20 12:56:31,435 DEBUG [M:0;jenkins-hbase20:40401] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-20 12:56:31,435 INFO [M:0;jenkins-hbase20:40401] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=344.96 KB heapSize=393.27 KB 2023-04-20 12:56:31,531 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-20 12:56:31,531 INFO [RS:0;jenkins-hbase20:36277] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase20.apache.org,36277,1681995213947; zookeeper connection closed. 2023-04-20 12:56:31,531 ERROR [Listener at localhost.localdomain/42205-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@61a8a9f9 rejected from java.util.concurrent.ThreadPoolExecutor@5962b61e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2678] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-20 12:56:31,531 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): regionserver:36277-0x100ce703e210001, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-20 12:56:31,531 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@88968d] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@88968d 2023-04-20 12:56:31,531 ERROR [Listener at localhost.localdomain/42205-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@428583ad rejected from java.util.concurrent.ThreadPoolExecutor@5962b61e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2678] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-20 12:56:31,532 INFO [Listener at localhost.localdomain/42205] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-04-20 12:56:31,846 INFO [M:0;jenkins-hbase20:40401] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=344.96 KB at sequenceid=573 (bloomFilter=true), to=hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/3dfc002ce4814e709caf3627adc26fe4 2023-04-20 12:56:31,850 INFO [M:0;jenkins-hbase20:40401] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 3dfc002ce4814e709caf3627adc26fe4 2023-04-20 12:56:31,850 DEBUG [M:0;jenkins-hbase20:40401] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/3dfc002ce4814e709caf3627adc26fe4 as hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/3dfc002ce4814e709caf3627adc26fe4 2023-04-20 12:56:31,854 INFO [M:0;jenkins-hbase20:40401] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 3dfc002ce4814e709caf3627adc26fe4 2023-04-20 12:56:31,854 INFO [M:0;jenkins-hbase20:40401] regionserver.HStore(1080): Added hdfs://localhost.localdomain:45293/user/jenkins/test-data/3c925716-d636-8fe8-b2ac-2abced4a70dd/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/3dfc002ce4814e709caf3627adc26fe4, entries=65, sequenceid=573, filesize=24.7 K 2023-04-20 12:56:31,855 INFO [M:0;jenkins-hbase20:40401] regionserver.HRegion(2948): Finished flush of dataSize ~344.96 KB/353238, heapSize ~393.26 KB/402696, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 420ms, sequenceid=573, compaction requested=false 2023-04-20 12:56:31,856 INFO [M:0;jenkins-hbase20:40401] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-20 12:56:31,857 DEBUG [M:0;jenkins-hbase20:40401] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-20 12:56:31,865 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-20 12:56:31,865 INFO [M:0;jenkins-hbase20:40401] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-04-20 12:56:31,865 INFO [M:0;jenkins-hbase20:40401] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:40401 2023-04-20 12:56:31,867 DEBUG [M:0;jenkins-hbase20:40401] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase20.apache.org,40401,1681995213032 already deleted, retry=false 2023-04-20 12:56:31,968 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-20 12:56:31,968 INFO [M:0;jenkins-hbase20:40401] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase20.apache.org,40401,1681995213032; zookeeper connection closed. 2023-04-20 12:56:31,968 ERROR [Listener at localhost.localdomain/42205-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@71fa0aa9 rejected from java.util.concurrent.ThreadPoolExecutor@6685073a[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4487] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-20 12:56:31,968 DEBUG [Listener at localhost.localdomain/42205-EventThread] zookeeper.ZKWatcher(600): master:40401-0x100ce703e210000, quorum=127.0.0.1:56673, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-20 12:56:31,969 ERROR [Listener at localhost.localdomain/42205-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@12fd08c3 rejected from java.util.concurrent.ThreadPoolExecutor@6685073a[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4487] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-20 12:56:31,971 WARN [Listener at localhost.localdomain/42205] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-20 12:56:31,976 INFO [Listener at localhost.localdomain/42205] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-20 12:56:32,089 WARN [BP-429953862-148.251.75.209-1681995210614 heartbeating to localhost.localdomain/127.0.0.1:45293] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-20 12:56:32,090 WARN [BP-429953862-148.251.75.209-1681995210614 heartbeating to localhost.localdomain/127.0.0.1:45293] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-429953862-148.251.75.209-1681995210614 (Datanode Uuid ceb39797-2841-41e5-8250-6d3f85306e32) service to localhost.localdomain/127.0.0.1:45293 2023-04-20 12:56:32,095 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8f86655c-bc81-b89c-ce6f-c96a6f801026/cluster_69223e73-de8d-e90f-7d69-be39760cd8eb/dfs/data/data1/current/BP-429953862-148.251.75.209-1681995210614] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-20 12:56:32,096 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8f86655c-bc81-b89c-ce6f-c96a6f801026/cluster_69223e73-de8d-e90f-7d69-be39760cd8eb/dfs/data/data2/current/BP-429953862-148.251.75.209-1681995210614] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-20 12:56:32,132 INFO [Listener at localhost.localdomain/42205] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:0 2023-04-20 12:56:32,292 INFO [Listener at localhost.localdomain/42205] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-04-20 12:56:32,442 INFO [Listener at localhost.localdomain/42205] hbase.HBaseTestingUtility(1293): Minicluster is down